balena-os / balena-yocto-scripts

Helper scripts for balena yocto builds

Yocto tools for using with Balena

This repository provides helper scripts and tools for building Balena OS.

• build/barys: Used for native builds, barys is a wrapper script over bitbake that builds BalenaOS. Used to initialize a build directory and create device type json files out of the coffeescript files, and then run the default build. Use -n to just setup the build directory.
• build/balena-build.sh: Used to build in a container, this script downloads a container builder image and calls barys.
• automation/jenkins_build.sh: Used in jenkins automation to build the OS, requires a jenkins environment to work.

Contributing

Issues

For issues we use an aggregated github repository available here. When you create issue make sure you select the right labels.

Pull requests

To contribute send github pull requests targeting this repository.

Please refer to: Yocto Contribution Guidelines and try to use the commit log format as stated there. Example:

test.bb: I added a test

[Issue #01]

I'm going to explain here what my commit does in a way that history
would be useful.

Change-type: patch
Signed-off-by: Joe Developer <[email protected]>

Make sure you mention the issue addressed by a PR. See:

• https://help.github.com/articles/autolinked-references-and-urls/#issues-and-pull-requests
• https://help.github.com/articles/closing-issues-via-commit-messages/#closing-an-issue-in-a-different-repository

balena-yocto-scripts's Issues

Barys needs a clean build option

During the development of a new yocto build image, it is somentimes usefull to perform a clean build and ignore cached changes. Therefore I recommend to integrate the --clean-build flag similar to bitbake flags.

Reference:
-c clean
-c cleanall
Mega manual

barys: The development-image flag is broken by "update development image variables"

Summary

The --development-image flag has no effect because it is modifying a variable that does not exist in the TEMPLATECONF.

Description

The --development-image flag should modify the DEVELOPMENT_IMAGE variable in local.conf but instead is attempting to modify OS_DEVELOPMENT.

• balena-intel - meta-balena-genericx86 - local.conf.sample

Steps to reproduce

This has been reproduced on the following commit, but is present in other versions.

• branch: master
• repo: balena-intel
• commit bad651bb75b6e5c1c81254411ce16985ee75c79b
• tag: v2.83.18+rev4

YOCTO_DOWNLOADS="/mnt/downloads"
YOCTO_STATE_CACHE="/mnt/sstate-cache"

./balena-yocto-scripts/build/barys \
    -m 'genericx86-64-ext' \
    -k --rm-work --build-history --development-image \
    --shared-downloads "${YOCTO_DOWNLOADS}" \
    --shared-sstate "${YOCTO_STATE_CACHE}"

Caused by

• #239
• author: @markcorbinuk

tree 6767149740e887d4a759bcb7bf065f6771b4c9b5
parent b25459ffc7f5c8c22a6d5d7c3afd66d9dc691801
author Mark Corbin <[email protected]> Tue Oct 19 10:40:53 2021 +0000
committer Mark Corbin <[email protected]> Tue Oct 19 10:40:53 2021 +0000

barys: update development image variables

Update the DEVELOPMENT_IMAGE references to use OS_DEVELOPMENT
following the changes to the handling of OS variants.

Change-type: patch
Changelog-entry: barys: update development image variables
Signed-off-by: Mark Corbin [email protected]
diff --git a/build/barys b/build/barys
index 2ffaee1..8ed2bac 100755
--- a/build/barys
+++ b/build/barys
@@ -444,9 +444,9 @@ $SCRIPTPATH/generate-conf-notes.sh $SCRIPTPATH/../../layers/${BALENA_MACHINE_LAY
 export TEMPLATECONF=${SCRIPTPATH}/../../layers/${BALENA_MACHINE_LAYER}/conf/samples
 source ${SCRIPTPATH}/../../layers/poky/oe-init-build-env ${SCRIPTPATH}/../../${BUILD_DIR}
 if [ "x$DEVELOPMENT_IMAGE" == "xyes" ]; then
-    sed -i "s/.*DEVELOPMENT_IMAGE =.*/DEVELOPMENT_IMAGE = \"1\"/g" conf/local.conf
+    sed -i "s/.*OS_DEVELOPMENT =.*/OS_DEVELOPMENT = \"1\"/g" conf/local.conf
 else
-    sed -i "s/.*DEVELOPMENT_IMAGE =.*/DEVELOPMENT_IMAGE = \"0\"/g" conf/local.conf
+    sed -i "s/.*OS_DEVELOPMENT =.*/OS_DEVELOPMENT = \"0\"/g" conf/local.conf
     # Always activate resinhup bundles in production builds
     sed -i "s/.*RESINHUP ?=.*/RESINHUP ?= \"yes\"/g" conf/local.conf
 fi

Should also upload the device logo.svg along with the rest of the deployed files

we should update the image deployment scripts to also upload the device type's svg logo in the image release folder on S3. Initially we will include the logo under each release, so that we will need a minimal change to the current deployment process.
Eg: https://resin-production-img-cloudformation.s3.amazonaws.com/images/am571x-evm/2.0.0-beta12.rev1/logo.svg

HQ: https://github.com/balena-io/balena/issues/1818

`latest` file should be publicly accessible

The latest text file that contains the name of the latest OS version should be publicly readable, enabling a simple way to programmatically retrieve the most up to date OS version.
This line needs to be changed -->
https://github.com/resin-os/resin-yocto-scripts/blob/49130dd8f1d14a33b82875d5c528fa6790df31d9/automation/jenkins_build.sh#L343

https://www.flowdock.com/app/rulemotion/resin-devices/threads/qgUIi7CftRNEbHcUfFcfEbR90Uo

Should stop setting IGNORE files on S3 for release invalidation after 2022-12-23

[thgreasi] See: https://jel.ly.fish/milestone-stop-using-s3-ignore-files-invalidate-releases-b01d365

Question: What all those artifacts created by jenkins_build.sh?

Hi,
i have a question regarding the jenkins_build.sh
I successfully build an image and now have some artifacts lying around i don't know what to do with.
The artifacts generated by the scripts are:

./VERSION_HOSTOS
./VERSION
./kernel_source.tar.gz
./device-type.json
./image/balena.img.zip
./resin-image-raspberrypi3.manifest
./kernel_modules_headers.tar.gz

Now, i have no idea what do to with the following:

./VERSION_HOSTOS
./VERSION
./resin-image.docker
./device-type.json

Can anyone explain what's the purpose of those files in the context of building balenaOs?

Supervisor tag is ignored during jenkins build

See https://jenkins.dev.resin.io/job/yocto-raspberrypi3/692/ - I set "multicontainer" as supervisor tag, but the build reports v6.1.2 in VERSION.
Not sure if the problem is here or in meta-resin, or both.

Upload additional manifests to S3

For devices with a flasher image type we are only uploading the flasher manifest to S3, which doesn't help to extract things like supervisor or engine versions.

We should also upload the other related manifests when available, like rootfs, initramfs, etc.

Example tool to view manifests: https://balena-io-playground.github.io/balenaos-manifest/

Related discussion on FD: https://www.flowdock.com/app/rulemotion/r-beginners/threads/Yyim97zSU9lFvOoCtt-EzeXWO5a

Add ability to run tests if present in device repos

Allow for having tests embedded in the device repositories with automation to run them during jenkins build.

Proposed directory structure is to have a directory called tests in the device repository root and have jenkins run tests/start.sh

Use versioning for the docker image

The plan is simple: keep a version file for the docker image
It has to be bumped when the breaking changes are added to the dockerfile
It has to be read when building the docker image, and used to tag that image
It has to be read when using the docker image (to load the image with the corresponding tag).

As the file is available inside of this repo (that is used as a submodule) it should provide the seamless integration with the current process (as the submodule is updated the file will be updated, etc.)

Issues when using npm 3.8.0

npm install coffee-script --production
fails when using npm 3.8.0 as it follows:

/home/andrei/work/resin/resin-os/resin-edison
└── (empty)

npm WARN enoent ENOENT: no such file or directory, open '/home/andrei/work/resin/resin-os/resin-edison/package.json'
npm WARN resin-edison No description
npm WARN resin-edison No repository field.
npm WARN resin-edison No README data
npm WARN resin-edison No license field.
npm ERR! code 1

As well coffee-script should be pin-pointed to 1.10 .

resin/resin-img doesn't clean up the container after running it

Cleanup json files before recreating them

Example for edison:
It would be nice to cleanup all jsons before running barys (even barys can do that). Because for example:

• you are on edison master you run build and it generates the i7 json
• you sw to production, you build, it generates nuc json

So you will end up on production branch with a i7 json. Which confused me and might confuse others too.
This is the current production - it will change in the future.

barys log seems broken

When running barys now, the log file looks incomplete:

====================barys STOP========================                                                                                                                                                              
====================barys STOP========================                                                                                                                                                              
====================barys STOP========================

Build scripts should check for existence of npm, node and jq commands and report accordingly

Should set the correct mimetype on S3 for the logo.svg

Currently it uses the default (binary/octet-stream), which prevents browsers to consume the icon. It should beContent-Type: image/svg+xml.
See:

balena-yocto-scripts/automation/jenkins_build.sh

Line 325 in 52d2eea

deploy_to_s3() {

See: https://www.flowdock.com/app/rulemotion/resin-frontend/messages/2266266
HQ: https://github.com/balena-io/balena/issues/1818

Images uploaded even if the prepare script errors out

Even if this line throws an error:

balena-yocto-scripts/automation/jenkins_build.sh

Line 340 in 84c60eb

/usr/src/app/node_modules/.bin/ts-node /usr/src/app/scripts/prepare.ts

the script doesn't bail, as it's in a subshell.

Token not optional during build

While building using balena-build.sh it prints that the token is an optional argument. But it fails if it doesn't find one in ~/.balena/token
Feels a bit misleading - maybe editing the help text to clearly state that we need a token - either as a param or in ~/.balena/ would make sense

balena-yocto-scripts/build/balena-build.sh

Line 18 in 67cb61f

-t Balena API token (optional - private apps access)

we should use release.semver for setting the release version on balenaCloud

see balena-io/balena-sdk#1093

Question: Why not mount docker.sock to run docker in docker?

I have a question reagarding the Dockerfile_yocto-build-env and that prepare-and-run.sh script.
Why are you starting the docker daemon in the docker file instead of just installing docker and using the /var/run/docker.sock to run the docker in docker?

Please see https://itnext.io/docker-in-docker-521958d34efd

Yocto Dunfell supports new BitBake Syntax since August 2021

BitBake in release Dunfell supports the new : syntax from version 1.46. See:
OSSystems/meta-browser#543 (comment)

I don't know the implications of adding that to NEW_SYNTAX_RELEASES=("honister" "kirkstone") in build/barys.
Since there are now Dunfell releases with and without : syntax it could cause issues downstream. I still think it might be worth investigating.

Helper container images should match repository version

The automation scripts, like jenkins_build.sh use a series of helper docker images from the resin dockerhub repository:

• balena-push-env
• yocto-build-env
• ${MACHINE}-package-based-hostext
• resin-img

Apart from resin-img which is a mistery, the rest are built from Dockerfiles images that are also in the balena-yocto-script repository by the jenkins_build-containers.sh script. This script is run periodically on PR by https://jenkins.dev.resin.io/job/yocto-build-env/configure

The automation script are always using the helper container images from master, while it makes more sense if they would use the version that matches the balena-yocto-script repository revision being built.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• Update Lock file maintenance

Detected dependencies

dockerfile

automation/Dockerfile_balena-push-env

automation/Dockerfile_yocto-build-env

github-actions

.github/workflows/flowzone.yml

• product-os/flowzone master

.github/workflows/yocto-build-deploy.yml

• easimon/maximize-build-space fc881a613ad2a34aca9c9624518214ebc21dfc0c
• actions/create-github-app-token v1.10.3@31c86eb3b33c9b601a1f60f98dcbfd1d70f379b4
• actions/create-github-app-token v1.10.3@31c86eb3b33c9b601a1f60f98dcbfd1d70f379b4
• actions/checkout v4.1.7@692973e3d937129bcbf40652eb9f2f61becf3332
• actions/checkout v4.1.7@692973e3d937129bcbf40652eb9f2f61becf3332
• actions/upload-artifact v4.3.6@834a144ee995460fba8ed112a2fc961b36a5ec5a
• docker/login-action v3.3.0@9780b0c442fbb1117ed29e0efdff1e18412f7567
• aws-actions/configure-aws-credentials v4.0.2@e3dd6a429d7300a6a4c196c26e071d42e0343502
• actions/create-github-app-token v1.10.3@31c86eb3b33c9b601a1f60f98dcbfd1d70f379b4
• actions/checkout v4.1.7@692973e3d937129bcbf40652eb9f2f61becf3332
• actions/download-artifact v4@fa0a91b85d4f404e444e00e005971372dc801d16
• balena-os/leviathan f4e933231aca5561d5e059c77ba93c4506130f43
• balena-os/leviathan f4e933231aca5561d5e059c77ba93c4506130f43

npm

build/contracts/package.json

build/package.json

Building with a specific supervisor tag fails

I just tried doing a Jenkins build setting supervisorTag to a specific branch, which in turn calls barys with a --supervisor-tag option.

The build failed with this error in the logs:

NOTE: Running task 4329 of 4426 (/yocto/resin-board/build/../layers/meta-resin/meta-resin-common/recipes-containers/docker-disk/docker-disk.bb:do_compile)
NOTE: recipe docker-disk---r0: task do_compile: Started
ERROR: docker-disk---r0 do_compile: docker-disk: TARGET_REPOSITORY and/or TARGET_TAG not set.

Dynamically select production/staging credential for app creation

The code needs to use different credentials to create public apps in staging or productions.

Fix setting meta-balena-base label for ESR builds

When deploying an ESR release, the release is labelled with the latest meta-balena version. Currently, the code fetches the latest tag
in meta-balena and uses that, but for ESR branches the latest meta-balena tag is in the form M.m.x while the latest meta-balena version (and the meta-balena-base) hostapp label, should be a balena semver so that the UI can perform version comparisons.

generate-conf-notes.sh generates confusing text

Running a rpi2 build for example:

Resin specific targets are:
    resin-image

Raspberry Pi 2                 : $ MACHINE=raspberrypi2 bitbake resin-image
    resin-image

Raspberry Pi (v1 and Zero)     : $ MACHINE=raspberrypi bitbake resin-image

"Resin specific targets are:" should list all the specific resin targets for all the supported boards (unique entries). Then we list per machine bitbake commands.