Open 3D Foundation Build Special Interest Group (SIG) documents
License: Apache License 2.0
Meeting notes from the O3DE Meetings.
The full table of previous meetings is available here.
SIG-Build is investigating making changes to the AR (Automated Review) to prioritize lowering infrastructure costs while still working on improving the developer experience and reducing build times.
The purpose of this document is to initiate a discussion with the TSC (Technical Steering Committee) and all SIGs regarding the current state of the AR and how it can be updated to lower our infrastructure costs. Most of the discussion around the proposed options will remain at a high level with the implementation details to be outlined later.
The AR (Automated Review) refers to a set of jobs that perform automated testing for pull requests submitted to O3DE. These tests are required to pass before PRs are merged. A SIG maintainer can run the AR on a pull request after they've performed their code review.
Check as shown in PRs (under CI/jenkins/pr-merge):
Jenkins pipeline view (shown after clicking on the Details link):
The AR jobs are orchestrated by Jenkins Pipeline and are defined using a Jenkinsfile stored in the O3DE repo: https://github.com/o3de/o3de/blob/development/scripts/build/Jenkins/Jenkinsfile.
The Jenkins server and its build nodes are hosted in AWS. The build nodes utilize EC2 instances for the hosts and EBS volumes for the build workspace. When a pull request is submitted a pipeline is generated, however build nodes are not created until an AR run is triggered. After an AR run starts, the required platform instances are spun up, builds/tests are executed, and the instances are terminated at the end of the run. Jenkins then reports the results back to the pull request. The EBS volume is retained to cache artifacts for the next run. A daily automated task deletes the EBS volumes for merged and inactive PRs.
Using this setup allows us to prioritize faster build times. However to reduce our costs, onboard new platforms, and maintain our developer velocity we will be investigating the following options.
SIG-Build is currently investigating the following options to reduce our infrastructure costs while still improving the PR experience for developers.
The main reason to migrate to GitHub Actions is to take advantage of the lower cost infrastructure and allow contributors to easily run tests in their own fork.
We have already migrated quicker running tests (Validation) to GitHub Actions. The next section will review the challenges of migration our builds and tests as well.
GitHub actions provides free usage for public repos using their standard hosted runners. The main trade-off with the lower cost is the slower build times using GitHub's runners compared to our self-hosted ones using 16-core machines. Changes to the core engine code will result in 4-5 hour build times.
Details on the hardware specs for GitHub's standard hosted runners.
| Operating System | CPU | Memory | Storage |
|---|---|---|---|
| Windows/Linux | 2-core CPU (x86_64) | 7GB | 14GB SSD |
| MacOS | 3-core CPU (x86_64) | 14GB | 14GB SSD |
More details info on the hosted runners can be found here: GitHub-hosted Runners
GitHub-hosted runners only provide 14GB of storage (mounted on /mnt) for all supported platforms. There is also a root volume that has about 29GB of space remaining out of the 84GB total which is used for the pre-installed environment.
Proposed workaround:
In the default configuration, the provided storage is not enough to support the build workspace for O3DE. In order to increase the available space beyond 14GB, we need to uninstall un-required software and utilize the space in the root volume.
Note: This is not an officially supported workflow, but is a currently available workaround.
This GitHub action is currently being tested: https://github.com/marketplace/actions/maximize-build-disk-space
In addition to removing the pre-installed software, this GitHub action frees up space by performing the following steps (Note: This is only required for Linux nodes. For other platforms we can simply move the workspace to the root volume.):
This results in a workspace with about 50GB of hard drive space.
Below are some of the use-cases that will be compatible with GitHub's standard hosted runners.
These are checks that run on the code base that don't require high CPU usage or require large build artifacts. This includes code style checks, static code analysis, etc. A simple clone of the O3DE repo doesn't run into any of the space limitations.
Example use cases:
Daily/Weekly builds that are not as time sensitive as AR builds would be a good use case for the standard runners.
GitHub also provides larger spec runners (up to 64 cores) that can be added to GitHub action workflows. The specs and billing rates can be found here:
Comparison between GitHub Larger Runners and AWS EC2 Instances (using 16-core machines):
| Platform | AWS EC2 (Hourly) | GitHub Hosted Runner (Hourly) |
|---|---|---|
| Windows (16 core) | $1.35 | $7.68 |
| Linux (16 core) | $0.62 | $3.84 |
This option would be similar to the GitHub runners listed above, but would take advantage of any available build infrastructure hosted by the Linux Foundation. SIG-Build will need to investigate if there are resources that are available for O3DE.
Another method to lower our costs is to reduce the number of AR runs required per PR. This option can be implemented to lower costs with our current Jenkins infrastructure and can also be used to optimize the GitHub actions workflow discussed earlier. The focus here would be to batch PRs using merge queues, separate tests into smaller jobs, and make certain tests optional.
Using a merge queue reduces the total number of AR runs by batching approved PRs and testing those changes with a single AR. If the AR passes, those PRs are then merged in together. GitHub recently released their merge queue feature in limited public beta and there are other third-party solutions as well that are free for open source projects. The merge queue would reduce costs by reducing the total number of AR runs over time. However with this strategy we need to implement mechanisms to address AR failures.
Build Failures
One item SIG-Build is currently investigating is a mechanism to identify the owners of each build failure: https://github.com/o3de/sig-build/blob/main/rfcs/rfc-bld-20220623-1-build-failure-rca.md
This can also be used to identify the PR associated with the build failure, remove it from the queue, and restart the AR run. There are other strategies like bisecting the queue and rerunning the AR separately to identify the problematic PR, but this does come at an increased cost.
Another item we will need to address are un-reliable tests that typically require re-running the AR in order to pass. Failures like these in a merge queue setup will cause unnecessary merge queue runs and will frustrate developers.
This option is more of a workflow change and relies on maintainer to run jobs based on type of changes (e.g. small bug fix, core engine changes, etc.). Contributors can also confirm they've run the provided tests in their fork.
At the moment our test suites are executed in a single Jenkins job that typically tasks 40-50min to run for an incremental build and up to 2 hours for a clean build on a 16-core machine. Splitting up the tests will allow us to selectively run tests in a manual or automated way. This will also make it more feasible to run tests within the GitHub Actions workflow detailed above.
This RFC provides additional details on the deployment stage for the Jenkins pipeline outlined here: https://github.com/o3de/sig-build/blob/main/rfcs/rfc-19-jenkins-pipeline-repo.md
The purpose of this RFC is to have the deployment stage setup with two pipelines (we initially assumed a single pipeline). In this workflow, changes are tested in the staging branch/environment separately. To deploy the change to prod, a PR is created to merge the changes from the staging branch to the prod branch.
The main goal of the separate pipelines setup is to reduce the amount of PRs created for each deployment. By having a separate pipeline for staging, these changes can be tested before creating a single a PR for the deployment.
There are some caveats to setting up and deploying Jenkins that may complicate a single pipeline setup. Those issues are detailed below.
Updating plugins requires an additional level of testing in the staging environment. This is due to the complexity of the dependencies in Jenkins plugins. Here are some examples:
Upgrade the minumum supported Visual Studio Version to 16.11 as 16.7 and add support to Visual Studio 17.
Microsoft is going to remove support for Visual Studio 16.9 in October 2022 and recently has released a new version (2022) that is able to compile and run most of O3DE without any issue but it is left out as the Project Manager is requesting specifically fot 16.x.
vswhere invocation are differentInvestigate current issues with the DCO plugin for Github that block integration workflows, such as users changing their username after signing off
Create tasks to improve the current DCO plugin or replace it with a better alternative
I brought forward to the TSC, on the Discord meeting of 4/19/2022, the proposal
to enable MacOS & iOS builds in AR. The TSC suggested to bring the proposal to SIG-Build.
A very important corporation is committing 1.5 engineers to help the development of the Metal RHI.
We fear that if Mac/iOS is not validated in AR, then it will be constantly broken
and it will be a frustrating experience for those willing to contribute
to MacOS/iOS in O3DE.
Procure enough "dedicated host" instances on EC2 to support AR runs for Mac/iOS.
Ideally the M1 chip based mac instances (now in 'Preview'):
https://aws.amazon.com/ec2/instance-types/mac/
steps to enable in AR:
Monthly cost for 1 instace of mac2.metal (The ones with M1 chip)
M1 chip with 8 CPU cores, 8 GPU cores, 16 GiB of memory, 16 core Apple Neural Engine. 1TB of storage.
$892.99 USD / month
EC2 Dedicated Host cost (monthly) $790.59 USD / month
Elastic Block Storage (EBS) pricing (monthly) 1TB $102.40 USD / month
From AWS:
"They deliver up to 60% better price performance over x86-based EC2 Mac instances for iOS and macOS application build workloads".
"you can save up to 44% off On Demand pricing with a three-year commitment."
During peak hours the AR system may spawn 120 Windows-based EC2 instances in parallel.
Unfortunately, We can not request Mac EC2 instances on demand, instead the only option is to have "dedicated hosts".
For example reserving 30 Mac hosts would cost around $26,790 USD /month, before discounts.
After the 44% discunt mentioned above it would be ~$15,000 USD / month.
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
JT - Discussed interfacing with ASWF https://github.com/AcademySoftwareFoundation/wg-ci
JT - Better docs on build and customization of the engine
JT - Discuss on any Blender entry points in Python (can bring up in next SIG-Build meeting for triage)
The purpose of this feature is to provide the O3DE Jenkins Pipeline setup to developers. This includes maintainers/contributors that will submit improvements to the O3DE Jenkins pipeline and game developers that want to use the AR pipeline for their own project.
The goal is to move the pipeline configuration from an internal process to a public one managed by the Build SIG with all the required files stored in a repo under the O3DE org. The current setup has a combination of CloudFormation stacks and manually deployed resources managed by an internal Amazon team.
The release of this feature will not change the behavior of the AR pipeline, but it will allow the O3DE community to maintain and contribute changes to the Jenkins setup.
This is important because it allows developers to contribute changes to the pipeline setup and enables the following:
All the scripts and configs required to setup our Jenkins infrastructure will be packaged in a repo. A pipeline will automatically deploy the required resources when updates are merged in.
A CDK package will also be included to easily deploy the service to AWS. This includes a CodePipeline instance that will be used to build, test, and deploy the service.
Use cases:
Update the Jenkins server setup: a user will clone the repo and make changes (e.g. install a new plugin, update the jenkins version, etc.). Then they can test these changes locally by building the docker image and running the container on their local machine. After testing, the user can then submit a PR with their changes. After the changes are reviewed and merged, the pipeline will take the change, deploy it to staging, then finally to production.
Use the AR for game projects: a user will fork the repo and make any customizations they need to the Jenkins server setup. Using this configuration they can easily setup the AR pipeline to run on their project. They can host Jenkins on their own server or a cloud provider using the templates provided.
Jenkins config components:
scripts/build/Jenkins/Jenkinsfile in the O3DE repo.AWS Hosting components:
Deployment workflow:
The core part of this setup is the docker container used to run the Jenkins server. To build the image, we pull down the latest LTS version of the Jenkins docker image and run our Dockerfile with our customizations.
FROM jenkins/jenkins:2.319.2-lts-jdk11
Here are the options to test changes before they are deployed to production:
cdk synth to validate changes to the CDK packages prior to deployment and use cdk deploy to test the changes in their own AWS account.In the event that issues are identified in the prod stack, a rollback mechanism will be required to redeploy the last known good version.
These issues should be identified with automated mechanisms to rollback quickly and effortlessly. We should also provide a manual rollback mechanism if issue are identified outside of automated testing.
CDK Pipelines doesn't provide a built-in mechanism to rollback yet, however the CodeDeploy resource within the Jenkins ECS stack can be utilized to perform a rollback if necessary.
For simplicity, a manual approval step will be added to promote changes from staging to prod. We will need to investigate additional mechanisms to promote these changes. This includes maintenance windows, merge queues, etc. We will need to work with the build SIG to identify the users that will have access to promote changes.
Additionally, a subset of build SIG maintainers will need direct access to the AWS account hosting the pipeline. This will be required in the event an investigation requires access to the AWS account to resolve an issue.
Jenkinsfile in the o3de repo, developers can use this setup to run the same pipe on their project and customize as needed.An alternative is to migrate our pipeline workflows to GitHub actions. While we have plans to utilize this mechanism for smaller automated workflows, there are some limitations that prevent us from migrating the entire pipeline to GH actions. This is mainly due to storage and compute restrains for O3DE builds. It's possible to utilize self-hosted runners, so we would still maintain some of our current build infrastructure with this solution. That said, there are planned investigations into how we can integrate GH actions into our pipeline.
main will be created for our production deployments and branch protection will be enabled to require approvals and successful checks. The pipeline used for deployment will also have additional protections when moving changes from staging to prod.The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
1.
2.
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
1.
2.
List any additional items below!
Each pull request has to pass PR based build pipeline in order to be merged. This proposal is to reduce PR based build time by reusing more relevant build workspace from previous builds, and thus increase O3DE contributors' work efficiency.
O3DE Jenkins build workspace is located on EBS volume created from daily EBS snapshot. Currently, more than half of O3DE PR based builds take over 3 hours to finish, this is mainly because:
This is important because it reduces O3DE contributors' waiting time to merge pull requests and enables the following:
S3SIS (S3 Single Instance Storage) will be used to share build workspace between different build pipelines. It reduces the file transfer time and cost by only transferring the delta files. In addition, files with same content hash are deduplicated using S3SIS, this significantly saves S3 storage cost.
After each non-PR based AR build is done, the build workspace will be uploaded to S3 using S3SIS. PR based build will download and reuse the build workspace.
Use cases:
Due to the issue that CLI tool installed using O3DE Python cannot be executed, S3SIS CLI needs to be installed with system installed Python on build nodes.
Steps to install S3SIS CLI:
git clone https://github.com/aws-lumberyard/s3sis.git or download the source code.python setup.py install" to install S3SIS CLI.s3siscli configure" to configure S3SIS CLI.S3SIS CLI should be part of the build image, so it doesn't need to be installed and configured in every build.
S3SIS requires a label to upload and download, the label is used to group a set of file objects on S3.
In order to make Jenkins build be able to find the correct workspace to sync, the label needs to include information about pipeline name, platform, build configuration and commit id.
Label format used to upload/download workspace using S3SIS:
{pipeline_name}{platform}{build_configuration}_{commit_id}
S3SIS manifest file stores necessary file information, and it's the main file to lookup when doing upload or download.
{
"label":"O3DE_Windows_profile_e0e455742f0109e7dfdf6ea44d9ec875cd0848a9",
"filelist":{
"engine.json":{
"isfile":true,
"md5":"cb87aae4199e0f920fd4f275a38fbfba",
"size":"2920",
"atimestamp":"1667166054970941766",
"mtimestamp":"1667166040290092687",
"attribute":"33206"
},
"Code/Framework":{
"isfile":false
},
"Code/Framework/CMakeLists.txt":{
"isfile":true,
"md5":"25af1ce1496b0b0fa9a0fcbec3c383bf",
"size":"576",
"atimestamp":"1667166810450666423",
"mtimestamp":"1667166810438665729",
"attribute":"33206"
}
}
}
To reduce the number of workspace uploaded, workspace is uploaded by non-PR based AR builds on success, like https://jenkins.build.o3de.org/job/O3DE/job/development/. The upload won't increase any PR build time.
If the build has multiple commits, s3siscli upload should run for each commit id. After first upload, all files are already on S3, the following upload command will upload nothing but a manifest file.
Upload workflow:
currentBuild.changeSets to find commit ids built in current build.s3siscli upload --label {pipeline_name}_{platform}_{build_configuration}_{commit_id} for each commit id.Most build systems use timestamp to decide when to rebuild. In order to reuse the build workspace, all files' timestamps and attributes should be preserved during the download process.
Ninja is used for Linux build, it tracks file timestamp at nanoseconds level. Ninja will rebuild a file whenever the file timestamp changed at nanoseconds level. In this case, all files' timestamps should be preserved at nanoseconds level.
Only first PR based build will sync workspace from the closest commit id, all following builds will reuse workspace from previous builds.
Download workflow:
{pipeline_name}_{platform}_{build_configuration}_{parent_commit_id}.--label {label} --preserve-timestamp --preserve-attributes --preserve-empty-folders --cleanup to syn workspace.Stale files should be cleanup regularly to avoid high S3 storage cost.
Each file on S3 is linked to one or multiple S3SIS manifest file, and each manifest file is linked to a commit id. A manifest is considered to be stale if the commit is over 1 month old.
Use a DynamoDB table to keep track of the number of manifest that each file is linked to. Set object hash as primary key to reduce lookup time. For example:
| object_md5_hash (Primary Key) | ref_count |
|---|---|
| cb87aae4199e0f920fd4f275a38fbfba | 3 |
| 25af1ce1496b0b0fa9a0fcbec3c383bf | 3 |
Run a daily lambda function to lookup manifest files, update the table and delete objects with 0 ref_count. Run a weekly lambda function to make sure S3 object, manifest files and DynamoDB table are synced.
Daily Cleanup Lambda Workflow:
The cost increased is mainly from S3 storage and S3 transfer. However, it reduces EC2 cost when build takes less time.
To reduce cost in the first place, only enable this feature for bottleneck build (Linux) since it directly impact the overall pipeline time. This feature can be rolled out gradually if it produces a good result.
S3 Cost Increased:
Linux build workspace is 180GB. Assume there are 150 AR builds and 200 pull request created per month.
In worst case scenario, all 180GB files are transferred for every build, the S3 cost would be $1341 per month.
Because S3SIS only transfers delta files, the files transferred will be significantly reduced, assume there are 30GB files transferred per build on average (in fact, the size could be less, it transfers 0 files for a zero build). The cost would be $223 per month.
EC2 Cost Reduced:
Linux node type is c4.4xlarge whose price is $0.796 per hour.
Assume 1 hour is saved on average for 1 PR based Linux build, 200 PR based builds would save $159.2.
First integrate this with O3DE bottleneck build, like Linux profile_nounity build, because it directly impacts the overall PR based build time. If it produces good result, then gradually roll this out to other builds.
Yes, using distributed build or build cache can also reduce overall build time.
Please fill any info related to the below for the 11/30 release notes: Note this has a due date of Friday Nov 12th, 2021
The purpose of this RFC is to document a proposed improvement to our pull request workflow using GitHub actions. This update will allow maintainers to manage PR workflows easily and minimize the time required to review and merge PRs. For example, an initial planned improvement is to allow maintainers to trigger the Automated Review (AR) for a PR by adding a comment and avoid having to navigate to another site to locate the job for that PR.
There are other workflow improvements that can be added through labels, comments, etc. Those specific workflow improvements will be out of scope of this RFC. This page will focus on the Github actions mechanism that will be used to implement these improvements.
This feature will utilize GitHub actions to execute the automation requests. Scripts will be added to the .github directory that are required for the workflows. There will be minimal infrastructure that the sig-build will need to maintain. Github's hosted runners will run the workflows and submit the requests to our existing infrastructure if required (e.g. Jenkins server).
Other workflow examples include adding a label once the review is completed so that the PR is merged once the AR is successful.
The main goal is to minimize the time required by maintainers to review and merge in PRs.
Workflow details
Setting a GitHub action trigger allows us to select the events that we will use to run our GitHub action. These events are typically specified in the "on:" block in the workflow yaml file.
on:
issue_comment:
types: [created]
issue comment will trigger all events for each comment (add, edit, delete) so we will filter to only created comments under types.
Also the "issue_comment" event occurs for comments on both GitHub issues and pull requests. So we'll need to filter for PR comments with the following.
if: ${{ github.event.issue.pull_request }}
The end result would look similar to this:
name: GitHub PR Workflow
on:
issue_comment:
types: [created]
jobs:
PR-Comment:
if: ${{ github.event.issue.pull_request }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v3
with:
python-version: '3.9'
- run: python run-ar.py
Within the Github workflow we can add custom actions (e.g. actions/setup-python@v3) to perform certain jobs. To actually trigger the AR run we'll utilize a script we'll have in the repo (e.g. run-ar.py) and run this script to perform all the required actions.
This service allows us to store sensitive information that our GitHub action requires as part of the execution. For example, we'll need to credentials to submit the AR run request to Jenkins.
The Jenkins server referenced in this RFC will be the O3DE server (https://jenkins.build.o3de.org/) and will be the target of the AR run requests.
The same permissions used to grant access to run the AR and merge PRs will be used in this automated workflow. For example, only users that have permissions to run the AR will be able to add a comment that actually triggers the AR. This also applies to members that are part of child teams of the larger maintainer/reviewer teams.
These permissions checks can be performed in the GitHub action workflow itself or as part of the script execution.
Custom rate limits will be placed on the GitHub action workflow runs in order to prevent unnecessary runs triggered by unintentional or malicious means. Prior to executing each run we will add a step to check the number of workflow runs requested by the user. If it is over our set threshold, we will limit the requests and add a comment to the PR.
For workflows like triggering the AR, we can also check if there is a run already in-progress and if there are already additional runs in the queue. If there are AR runs queued above our threshold, we'll prevent adding more to the queue.
When the GitHub action workflow is rate limited due to too many requests a single comment will be added to the PR for visibility.
Comments added to PRs that are used to trigger the GitHub actions will be prefixed with "/" (e.g. /run-ar) The purpose is to make it easy to separate feedback comments from automation triggers.
Yes, we can utilize webhooks to trigger automations running on our own hosted machines or on AWS managed services. The option of using GitHub actions here allows the sig-build to easily maintain these workflows by updating the scripts in the .github directory without having to maintain configs for other services.
The O3DE contribution website pages and README docs will be updated to instruct maintainers and reviewers to utilize new automated workflows as they are added.
Q: Is there a way to de-dupe the command comments? There could be many submitted for some PRs.
A: This is up for debate as it may be useful to keep the command history in the PR. However, we will need to investigate ways to reduce the noise on the PRs for the command comments.
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics:
Triage backlog grooming: https://github.com/o3de/o3de/issues?q=is%3Aissue+is%3Aopen+label%3Asig%2Fbuild+label%3Aneeds-triage+no%3Aassignee
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Triage GitHub Issues assigned to Build SIG
Discuss agenda from proposed topics
Discuss the scope and any other discussion points from Build SIG members
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
1.
2.
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
1.
2.
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
/scripts/build @o3de/sig-build-reviewers @o3de/sig-build-maintainers
/scripts/commit_validation @o3de/sig-build-maintainers
/scripts/license_scanner @o3de/sig-build-maintainers
/scripts/signer @o3de/sig-build-maintainers
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
SIG-Build is currently investigating how we can reduce the costs of the AR Pipeline.
Details are located here: #79
Since the inception of O3DE, each SIG chair has been staffed as an interim position. It's time to hold some official elections, following some of the proposed guidance but with our own process due to the holiday season and in order to expedite the elections into next year.
The chair and co-chair serve equivalent roles in the governance of the SIG and are only differentiated by title in that the highest vote-getter is the chair and the second-highest is the co-chair. The chair and co-chair are expected to govern together in an effective way and split their responsibilities to make sure that the SIG operates smoothly and has the availability of a chairperson at any time.
Unless distinctly required, the term "chairperson" refers to either/both of the chair and co-chair. If a chair or co-chair is required to perform a specific responsibility for the SIG they will always be addressed by their official role title.
In particular, if both chairpersons would be unavailable during a period of time, the chair is considered to be an on-call position during this period. As the higher vote-getter they theoretically represent more of the community and should perform in that capacity under extenuating circumstances. This means that if there is an emergency requiring immediate action from the SIG, the chair will be called to perform a responsibility.
Additionally, at this stage of the project, the SIG chairpersons are expected to act in the Maintainer role for review and merge purposes only, due to the lack of infrastructure and available reviewer/maintainer pool.
... And potentially more. Again, this is an early stage of the project and chair responsibilities have been determined more or less ad-hoc as new requirements and situations arise. In particular the community half of this SIG has been very lacking due to no infrastructural support, and a chairperson will ideally bring some of these skills.
Nomination may either be by a community member or self-nomination. A nominee may withdraw from the election at any time for any reason until the election starts on 12/3.
For this election, nominees are required to have at minimum two merged submissions to http://github.com/o3de/o3de (must be accepted by 2022-01-31). This is to justify any temporary promotion to Maintainer as required by this term as chairperson. Submissions may be in-flight as of the nomination deadline (2021-12-08 12PM PT), but the nominee must meet the 2-merge requirement by the end of the election or they will be removed from the results.
Any elected chairperson who does not currently meet the Maintainer status will be required to work with contributors from the SIG to produce an appropriate number of accepted submissions by January 31, 2022 or they will be removed and another election will be held.
The only other nomination requirement is that the nominee agrees to be able to perform their required duties and has the availability to do so, taking into account the fact that another chairperson will always be available as a point of contact.
Nominations will be accepted for 1 week from 2021-12-01 12:00PM PT to 2021-12-08 12:00PM PT.
Nominate somebody (including yourself) by responding to this issue with:
The election will be conducted for one week from 2021-12-08 12:00PM PT and 2021-12-15 12:00PM PT and held through an online poll. Votes will be anonymous and anyone invested in the direction of O3DE and the SIG holding the election may vote. If you choose to vote, we ask that you be familiar with the nominees.
If there is a current interim chair, they will announce the results in the Discord sig channel as well as the SIG O3DE mailing list no later than 2021-12-17 1:00PM PT. If there is no interim chair, the executive director will announce the results utilizing the same communication channels. At that time if there is a dispute over the result or concern over vote tampering, voting information will be made public to the extent that it can be exported from the polling system and the SIG will conduct an independent audit under the guidance of a higher governing body in the foundation.
The elected chairpersons will begin serving their term on 2022-01-01 at 12AM PT. Tentatively SIG chairs will be elected on a yearly basis. If you have concerns about wanting to replace chairs earlier, please discuss in the request for feedback on Governance.
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
1.
2.
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
1.
2.
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Updates incoming for the Jenkins server instance. Plugins are being updated to latest. Will be deployed at EoD today (April 20th)
Codeowners additions for SIG-Build
-Triage the new issues in the queue - https://github.com/o3de/o3de/issues?q=is%3Aissue+is%3Aopen+label%3Asig%2Fbuild+label%3Aneeds-triage+no%3Aassignee.
/scripts/build @o3de/sig-build-reviewers @o3de/sig-build-maintainers
/scripts/commit_validation @o3de/sig-build-maintainers
/scripts/license_scanner @o3de/sig-build-maintainers
/scripts/signer @o3de/sig-build-maintainers
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
1.
2.
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics:
Triage backlog grooming: https://github.com/o3de/o3de/issues?q=is%3Aissue+is%3Aopen+label%3Asig%2Fbuild+label%3Aneeds-triage+no%3Aassignee
List any additional items below!
The purpose of this RFC is to document the build SIGs intent to migrate the nightly AR (Automated Review) pipeline over to O3DE's build infrastructure and to highlight the blockers for certain platforms.
This effort will require an investigation into which platforms we want to migrate and support in the nightly pipeline prior to starting this work.
Here are the main goals of migrating the pipeline over to O3DE:
Note: There are also nightly pipelines created for stabilization branches prior to a release.
This section contains details on the platforms that we cannot migrate in their current state.
| Platform | Blockers |
|---|---|
| Mac/iOS | Costs will exceed budget. Autoscaling currently not supported. Dedicated hosts are required to deploy Mac EC2 instances which are billed while the hosted instance is shutdown. Minimum 24hr allocation. |
| Windows GPU | Costs will exceed budget. Dedicated hosts not required, so it's possible to autoscale. A solution is required to auto-connect instances when they startup. |
The main strategy of the migration plan is to transfer the nightly jobs to O3DE in phases. This will allow us to meet the goals identified above while we address the blockers.
| Task | Description | GitHub Issue/PR |
|---|---|---|
| Identify platforms to migrate in the first pass | First we need to identify the platforms that we want to support in the nightly pipelines, so that we can make the plan to migrate in phases. Check with the required SIGs for approval. | Completed |
| Split up nightly pipeline into separate tags | Update the build_config.json file for each platform to separate the nightly pipeline using the platforms identified in the previous task. This will allow us to run the excluded platforms in an internal pipeline while we address the blockers. | o3de/o3de#9406 |
| Create Jenkins pipelines | Create nightly pipelines on O3DE Jenkins instances to run the separate nightly pipelines created in the previous task. | o3de/o3de#9427 |
| Create plan to address Mac/iOS blockers | Investigate solutions to address the blockers to migrate Mac/iOS platforms to O3DE. This includes evaluating other mac hosting services. | o3de/o3de#9428 |
| Create plan to address Windows GPU blockers | Investigate solutions to address the blockers to migrate the Windows GPU platform to O3DE. | o3de/o3de#9429 |
The proposal here is to update the build_config.json files across the supported platforms to use the cmake-presets to configure configuring and building a project.
The cmake-presets option can also be used to configure the options for running CTest as well.
With CMake 3.23 adding support for an include option to specify other presets files to include inside the main CMakePresets.json, we can take advantage of it to provide separate per platform preset files to configure, build and run CTest for each of the platforms we support.
Using the cmake-presets options allows O3DE to reconcile the options that users follows to configure using cmake with how the Jenkins build server.
When a user wants to configure and build using cmake, they this specifying the options directly to the cmake command.
An example of a user configuring cmake is as follows
# Configure and generate a build solution
cmake -B <build-dir> -S <source-dir> -G <generator> -DLY_3RDPARTY_PATH=<3rdPartyPath> ...`
cmake --build <build-dir> --config <config>
Now when the Jenkins server performs a build, it runs the ci_build.py script to configure and kick offs a build as part of its pipeline steps
python\python.cmd -u scripts\build\ci_build.py --platform Windows --type profile
The cmake-presets option provides a standardize CMake way to specify options to the CMake configure, build and test steps.
We have previously discussed crated a CMakePresets.json file in a discussion in July 2021.
Users and the CI pipeline would use this feature via the cmake --preset option.
Here is a psuedo example of how to configure for O3DE for Windows.
cmake --preset Windows <[additional options if needed>]
The cmake --build command supports the preset option as well.
cmake --build --preset Windows <[additional options if needed>]
Finally the CTest also supports the --preset option as well
ctest --preset SUITE_main
ctest --preset SUITE_smoke
This works would involve adding a CMakePresets.json file to the o3de repo as a well as in the project templates root as well.
The CMakePresets file would be setup to use the new CMake 3.23 preset include feature other platform specific presets files from the <repo-root>/cmake/Platform/<Platform> directory.
This allows each platform that can be built on the host platform to be additive with the list of presets available for use.
Here is a sample of a how the CMakePresets.json inside the O3DE repo root could look
{
"version": 4,
"cmakeMinimumRequired": {
"major": 3,
"minor": 23,
"patch": 0
},
"include": [
"cmake/Platform/Android/CMakePresets.json",
"cmake/Platform/iOS/CMakePresets.json",
"cmake/Platform/Linux/CMakePresets.json",
"cmake/Platform/Mac/CMakePresets.json",
"cmake/Platform/Windows/CMakePresets.json"
],
"configurePresets": [
{
"name": "default",
"displayName": "Default Config",
"description": "Default configuration which builds with Unity. Inherits Generator and Binary directory",
"cacheVariables": {
"LY_UNITY_BUILD": {
"type": "BOOL",
"value": "ON"
}
},
"inherits": "${hostSystemName}-default"
}
],
"buildPresets": [
{
"name": "default",
"configurePreset": "default"
}
],
"testPresets": [
{
"name": "default",
"configurePreset": "default",
"output": {
"outputOnFailure": true
},
"execution": {
"noTestsAction": "error"
},
"filter": {
"include": {
"label": "(SUITE_smoke|SUITE_main)"
},
"exclude": {
"label": "(REQUIRES_gpu)"
}
}
}
]
}The per platform CMake Preset json files would add in configure and build presets that provides the defaulkt generators and build directories a customer can use.
Authoring presets to use to the CMake 3.23 include features will NOT break current workflows. This is purely an opt-in feature.
Developers would interact with the system using the --preset option when
Well not implementing CMake presets for O3DE is an option.
O3DE currently has it's own psuedo preset system in the form of using the ci_build.py script to invoke a build configuration listed in a build_config.json.
The issue with that is that is not standard, not documented within o3de and not integrated with CMake.
Due to that it is not easy for a user to discover that they can us the ci_build.py script to normalize their builds and they wouldn't have any known commands to run to list which build steps are available
An Impactful change notice will be posted to users announcing that O3DE supports CMakePresets and that the list of available presets can be listed using the cmake --list-presets command.
A documentation page on o3de.org will also be added to list examples of using a CMakePreset for configuring, building and running a test.
Users would be able to use the official CMake documentation to learn how to create their own CMake Presets: https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html
How many presets should O3DE expose out of the box? Every preset that is added to the CMakePresets.json is that has to receive a level of support by the O3D engineers and caution would needs to be exercised to prevent adding too many presets.
The more presets there are the higher the likelihood that one would need to be removed which can strain user expectations when they are using such a preset. On top of that the more presets there out, the higher the cognitive burden a user would have to go through to discover which ones to use.
When should the Jenkins build nodes be updated to CMake 3.23 and the build scripts updated to invoke these presets.
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics:
Triage backlog grooming: https://github.com/o3de/o3de/issues?q=is%3Aissue+is%3Aopen+label%3Asig%2Fbuild+label%3Aneeds-triage+no%3Aassignee
List any additional items below!
This RFC the details the solutions to provide security scanning for the O3DE Jenkins Pipeline repo.
The main objective is to provide a mechanism to improve the security and best practices of the contributions to the cloud infrastructure and docker configurations in this repo.
The components we want to scan in the repo are the CDK packages and Docker images. All contributions will be required to pass the configured checks.
The scope of the security scanning is limited to the Jenkins server infrastructure and the components used to deploy it. This does not include the build node configs. Those configs are stored in the o3de repo: Build Node Configs
The two main components that make up the O3DE Jenkins Pipeline repo are the CDK packages and the Docker image setup.
| Component | Usage | Environment |
|---|---|---|
| CDK Packages | Defines and deploys the AWS Infrastructure to host the Jenkins pipeline server. | Python |
| Docker | Creates the image for the container that runs the Jenkins pipeline server. | Linux shell commands |
The CDK scanning solutions typically provide coverage using one of these two options:
The tools that scan the CDK constructs will evaluate the code within each of the CDK packages for any potential security vulnerabilities that will created in the final CloudFormation stack. The benefit of scanning CDK constructs is that the static analysis and synthesis steps can be performed at the same time during the cdk synth step. Developers can generate the CF templates and run the static analysis with a single command. However, this does require adding an additional construct within the CDK app to support this.
The other options scan the generated CloudFormation templates after the packages are created using cdk synth. Scanning the generated templates does not require any changes to the codebase and these scanners are typically stand-alone applications.
Cdk-nag is an open source tool developed by AWS that provides the ability to scan CDK packages for security and best practice violations. To enable the checks, a cdk-nag construct needs to be added either to the top-level app or the individual stacks.
Example adding cdk-nag to the top-level app. This will enable the checks when running cdk synth or cdk deploy.
[...]
import cdk_nag
app = cdk.App()
Aspects.of(app).add(cdk_nag.AwsSolutionsChecks())
Example output:
cdk synth
[Error at /S3BucketWithCfnNagConstructStack/S3Bucket/Resource] AwsSolutions-S1: The S3 Bucket has server access logs disabled.
[Error at /S3BucketWithCfnNagConstructStack/S3Bucket/Resource] AwsSolutions-S2: The S3 Bucket does not have public access restricted and blocked.
[Error at /S3BucketWithCfnNagConstructStack/S3Bucket/Resource] AwsSolutions-S3: The S3 Bucket does not have default encryption enabled.
This tool is provided by Docker and provides the ability to scan the generated docker image for security vulnerabilities. This can run locally and in our pipeline, but does require that we first build the image prior to each scan.
Example setup and scan commands:
# Setup
apt-get install docker-scan-plugin
# Build image
docker build -t jenkins .
# Run scan
docker scan jenkins
In addition to the options detailed above, this section will provide scanning options for other components in the O3DE Jenkins Pipeline repo.
Dependabot
Dependabot can be used to keep the dependencies defined in the repo up-to-date. Dependencies are defined in the following locations:
Dependabot supports updating both docker and python dependencies. Enabling this tool in our repo can be found in its GitHub settings: Repo > Settings > Code security and analysis
To complete the setup, a config file also needs to be added to the repo at the following location: .github/dependabot.yml
Documentation on the .yml file can be found here: Configuring dependabot version updates
Github Actions/Pull Request checks
Yes. Static analysis for CDK packages is still limited, so most of the other solutions scanned the generated CloudFormation templates instead of scanning the cdk code directly.
For Docker images, options to scan dockerfiles/shell commands directly used paid or restrictive licenses.
There are options for the AR pipeline that are not located in the Jenkinsfile. These options should be called out in the docs.
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
Notes from last meeting on June 15th: https://github.com/o3de/sig-build/blob/main/meetings/notes/sig-meeting-20220615.md
Brian - Followup on the GHI on mac and ios builds for nightly with shirang
Shirang - Comment on the RFC for intermittent tests on any existing work from sig-build and reach out to sean on periodic build failure concerns
Brian - Update Github PR RFC following feedback
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
1.
2.
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
/scripts/build @o3de/sig-build-reviewers @o3de/sig-build-maintainers
/scripts/commit_validation @o3de/sig-build-maintainers
/scripts/license_scanner @o3de/sig-build-maintainers
/scripts/signer @o3de/sig-build-maintainers
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
1.
2.
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
/scripts/build @o3de/sig-build-reviewers @o3de/sig-build-maintainers
/scripts/commit_validation @o3de/sig-build-maintainers
/scripts/license_scanner @o3de/sig-build-maintainers
/scripts/signer @o3de/sig-build-maintainers
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Introductions
Facilitator Deepak Sharma, Amazon, SDM
Participants Alphabetically - <Name, Company, Team Role>
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
1.
2.
List any additional items below!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
Build SIG Charter
This charter adheres to the Roles and Organization Management specified in "<sig-governance.md>".
Team information may be found in the "<readme.md>"
Overview of SIG
This SIG is the coordinating group that works with all sigs for O3DE Build and Infrastructure related activities in a cadence with a standard process. This SIG owns the systems that are used to build O3DE. Build SIG will also work on the build failure analysis and remediation, in addition to monitoring build times. In addition to that this SIG will also own the Infrastructure for O3DE including the systems and services related to Git, GitHub, Jenkins, GitHub Issues etc. This SIG is created with dedicated SIG contributors, reviewers and maintainers.
Responsibilities
Goals
Scope
In scope
Cross-cutting Processes
Out of Scope
SIG Links and lists:
Roles and Organization Management
Build SIG adheres to the standards for roles and organization management as specified by . This SIG opts in to updates and modifications to
Individual Contributors
Additional information not found in the sig-governance related to contributors.
Maintainers
Additional information not found in the sig-governance related to contributors
Additional responsibilities of Chairs
Additional information not found in the sig-governance related to SIG Chairs
Subproject Creation
Additional information not found in the sig-governance related to subproject creation
Deviations from sig-governance
Explicit Deviations from the sig-governance
We (sig-release) are cutting the stabilization branch on 4/4. By EOD 4/4 we will need the appropriate AR and build infrastructure setup for the branch. We will also need appropriately signed nightly installer builds setup as well.
This policy proposal documents O3DF's support levels for major dependencies that are maintained in the Automated Review (AR) system.
SIG-Build will maintain dependencies for the AR build nodes. Feature requests and bug fixes can be sent via Github and assigned to the SIG through labels or directly in the SIG-Build repo.
These are currently AWS EC2 AMI (Amazon Machine Image) that contain dependencies which include, but not limited to the following:
In conjunction with SIG-Platform, the general policy will be:
| OS | Version | Environment | Notes |
|---|---|---|---|
| Windows | 2019 1809 (eqv Windows 10 20H2) | AR | |
| Windows | 2022 21H2 (eqv Windows 11 21H2) | Nightly | Planned |
| Ubuntu | 20.04 LTS | AR | |
| Ubuntu | 22.04 LTS | Nightly | Planned |
| MacOS | Big Sur | AR | |
| MacOS | Monterey | Nightly | Planned |
| Build System | Version | Compiler | Version | OS | Environment |
|---|---|---|---|---|---|
| Visual Studio | 2022 17.3.0 | MSVC | v143 | Windows 2019 | AR |
| Visual Studio | 2019 16.9.2 | MSVC | v142 | Windows 2019 | Nightly |
| Ninja | 1.10.0 | Clang/LLVM | 12 | Ubuntu 20.04 | AR |
| Ninja | 1.10.1 | Clang/LLVM | 14 | Ubuntu 22.04 | Nightly |
| Ninja | 1.10.0 | GCC | 9.3.0 | Ubuntu 20.04 | Nightly |
| Ninja | 1.10.1 | GCC | 11.2.0 | Ubuntu 22.04 | Nightly |
| XCode | 13.2 | Clang/LLVM | 12 | MacOS Big Sur | AR |
| XCode | 13.4 | Clang/LLVM | 13 | MacOS Monterey | Nightly |
| SDK Type | Package Versions | OS | Environment |
|---|---|---|---|
| Android SDK | "platforms;android-28" "platforms;android-29" "platforms;android-30" | Windows 2019 | AR and Nightly |
| Android NDK | '"ndk;21.4.7075529"' | Windows 2019 | AR and Nightly |
| Android Google Play Packages | '"extras;google;market_apk_expansion" "extras;google;market_licensing"' | Windows 2019 | AR and Nightly |
| Android Build Tools | '"build-tools;30.0.2" | Windows 2019 | AR and Nightly |
| Tool Type | Package Versions | OS | Environment |
|---|---|---|---|
| CMake | 3.24.0 | Windows 2019 | AR and Nightly |
| CMake | 3.24.0 | Ubuntu 20.04 | AR and Nightly |
| CMake | 3.24.0 | Ubuntu 22.04 | AR and Nightly |
| CMake | 3.24.0 | MacOS Big Sur | AR and Nightly |
Build failure RCA system currently in use is to identify and categorize root causes of Jenkins builds failures. It depends on Jenkins Build Failure Analyzer plugin and search for certain log patterns from the build log.
However, current build failure RCA system has some bottlenecks due to the use of Build Failure Analyzer plugin:
The purpose of this RFC is to propose a solution to deprecate the use of Build Failure Analyzer plugin and solve the above bottlenecks.
This is important because O3DE contributors will have more control over how build failure root causes are identified, and this can be used in below scenarios:
Instead of using Build Failure Analyzer plugin to do build failure RCA on Jenkins server, use lambda functions to retrieve build log of each build platform/configuration and do build failure RCA.
A CDK package is used to deploy build failure RCA lambda functions, and build failure RCA patterns will be stored in the same CDK package. A CI/CD pipeline (Jenkins pipeline or CodePipeline) will be setup to verify and deploy the CDK package on every CDK change.
Use cases:
Build failure RCA system is deployed by CDK, it can be easily deployed to different environments.
Build failure RCA CDK Stack consists of following AWS resources:
Post Build SNS Topic: SNS topic is used to receive post build message from AR/Periodic builds, and it feeds the data to SQS queue.
SQS Queue: In case SNS topic reaches rate limiting and discard unprocessed data, SQS queue is used to receive post build message from Post Build SNS Topic, and it triggers Lambda function to do build failure RCA.
RCA Dispatcher Lambda Function: RCA dispatcher lambda function is triggered by post build SNS topic, it retrieves AR/Periodic build data and dispatch the actual RCA work for each build platform/configuration to separate lambda functions. Once it received the RCA result from RCA worker lambda function, it uploads the RCA result to CloudWatch. If notification is enabled, it will send build failure notifications via email, Discord or Slack.
RCA Worker Lambda Function: RCA worker lambda function retrieves build log of a certain build platform/configuration, it scans the build log and search for log messages that match build failure RCA patterns, once the RCA work is done, it uploads the build log to S3 and return RCA result to RCA dispatcher lambda function.
Build Log S3 Bucket: Build log S3 bucket is used to store build logs grouped by build platform/configuration.
CloudWatch Metrics: CloudWatch metrics is used to store build failure RCA result.
The CDK deployment pipeline is a Jenkins pipeline that deploys build failure RCA CDK stack on build failure RCA repo change events.
Every build failure RCA repo change must go through pull requests. Pull requests will be tested by CDK deployment pipeline that is hooked with the PR status check rule. The CDK deployment pipeline will deploy a temporary CDK stack for the pull request and run some tests, once the tests are done, the CDK stack will be destroyed.
AR/Periodic build log is aggregated by log messages from multiple build platforms/configurations, the only way to re-group the log by platform/configuration is to use BlueOcean API.
First the RCA dispatcher lambda function send a GET request
https://{JENKINS_URL}/blue/rest/organizations/jenkins/pipelines/{JOB_NAME}/runs/{BUILD_NUMBER}/nodes?tree=id,displayName,result,type,edges[*]
"tree" parameter is used to reduce Jenkins server load and reduce response size.
Response of the GET request contains the pipeline stage data.
{
"_class" : "io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeImpl",
"displayName" : "Android [release]",
"id" : "116",
"result" : "SUCCESS",
"type" : "PARALLEL",
"edges" : [
{
"_class" : "io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeImpl$EdgeImpl",
"id" : "778",
"type" : "STAGE"
}
]
},
{
"_class" : "io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeImpl",
"displayName" : "Setup",
"id" : "778",
"result" : "SUCCESS",
"type" : "STAGE",
"edges" : [
{
"_class" : "io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeImpl$EdgeImpl",
"id" : "1112",
"type" : "STAGE"
}
]
},
{
"_class" : "io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeImpl",
"displayName" : "release",
"id" : "1112",
"result" : "SUCCESS",
"type" : "STAGE",
"edges" : [
{
"_class" : "io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeImpl$EdgeImpl",
"id" : "1718",
"type" : "STAGE"
}
]
},
The RCA dispatcher lambda function uses these pipeline stage data to build a relationship graph of build platform/configuration, and then it triggers the RCA worker lambda function to do build failure RCA for failed build platform/configuration.
RCA worker lambda function sends a GET request
https://{JENKINS_URL}/blue/rest/organizations/jenkins/pipelines/{JOB_NAME}/runs/{BUILD_NUMBER}/nodes/{ID}/log
to get the build log of a certain build platform/configuration.
RCA worker lambda function scans the build log and search for log messages that match build failure RCA patterns.
Build failure RCA pattern is in below JSON format:
Example of a RCA pattern that matches single line, and it will find all occurrences.
{
"indications": [
{
"name": "Asset Processing",
"description": "Asset Processing error",
"comment": "Asset Processing error, see log for more details",
"single_line": true,
"find_all": true,
"log_patterns": [
"^.*AssetProcessor: JOB LOG:.*ERROR.*$"
],
"log_testcases": [
"AssetProcessor: JOB LOG: 5/26/2021 10:11 PM | ERROR"
],
"owners": [
"@asset_processing"
]
}
]
}
Example of a RCA pattern that matches multi line, and it will stop searching after first occurrence is found.
{
"indications": [
{
"name": "CTest failure(s)/error(s)",
"description": "Errors while running CTest",
"comment": "Errors while running CTest",
"single_line": false,
"find_all": false,
"log_patterns": [
"The following tests FAILED:.*Errors while running CTest"
],
"log_testcases": [
"The following tests FAILED:"
],
"owners": [
"@ctest"
]
}
]
}
The build failure RCA CDK deployment pipeline will be created on O3DE Jenkins and the CDK stack will be deployed to O3DE AWS account.
Yes, an alternative is to write a Jenkins plugin that can offload build failure RCA work from Jenkins server, and do separate build failure RCA for each pipeline stage.
Hi! SIG release is working to help the O3DE community to get visibility of the O3DE roadmap. Full context can be read here: o3de/sig-release#79.
In order to achieve that goal, we need your help to create the roadmap for your SIG by February 6th, 2023, and start giving brief updates about the roadmap items at the Joint TSC meeting on February 28th, 2023. Instruction to create the roadmap view can be read in the RFC section "Roadmap Review in TSC monthly meeting".
Let me know if you have any concerns with the dates or any questions about the ask!
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
Discuss agenda from proposed topics
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
There's reports that links to development and PR branch jobs pages in Jenkins result in 503s (service unavailable) between March 25 and March 28. Job execution in progress is not blocked, but seems to be preventing contributors from executing or monitoring the status of their builds
The SIG-Build Meetings repo contains the history past calls, including a link to the agenda, recording, notes, and resources.
What happened since the last meeting?
/scripts/build @o3de/sig-build-reviewers @o3de/sig-build-maintainers
/scripts/commit_validation @o3de/sig-build-maintainers
/scripts/license_scanner @o3de/sig-build-maintainers
/scripts/signer @o3de/sig-build-maintainers
Discuss outcomes from agenda
Create actionable items from proposed topics
List any additional items below!
When a new O3DE installer is generated it should be tested to verify it installs a working version of O3DE, and uninstalls O3DE correctly.
An estimated cost of $62/mo based on
A new nightly Jenkins job runs and performs smoke tests on any new installer builds created in the development and release branches.
The results of these jobs are emailed to maintainers and available via the Jenkins website for O3DE installer builds.
A new Jenkins Job is created to run after the nightly installer jobs for development and release branches.
NOTE: "release branches" refers to the stabilization branches we generate (e.g. stabilization/2110rte). We do not need a job for the main branch because it will be identical to the stabilization branch.
The installer tests will be Jenkins jobs that are triggered after the existing nightly installer jobs.
Regular users should not need to be aware of this feature, maintainers would remove install builds that are failing, or not upload them at all.
What about other platforms?
Initial testing will be for the Windows platform only, future tasks will address testing other platform installers.
Will automation be designed to prevent failing installer builds from being uploaded and will be users be warned?
Currently only maintainers will be notified of the installer test status and it will be up to them to determine the correct next steps depending on the severity of the failed test(s).
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.