tianocore / edk2-pytool-extensions

Extensions to the edk2 build system allowing for a more robust and plugin based build system and tool execution environment

Tianocore Edk2 PyTool Extensions (edk2toolext)

Getting Started? Check out our documentation at https://www.tianocore.org/edk2-pytool-extensions!

This is a Tianocore maintained project consisting of command line and other python tools and extensions for building and maintaining an Edk2 based UEFI firmware code tree. Part of these tools include "invocables" that can be used to build EDK2 Platforms and these tools are known as "stuart". This package's intent is to provide tools, scripts, and a plugin based environment for use within the tools and scripts. This environment has command line interfaces to support building a product, building CI, running tests, and downloading dependencies. This environment also provides the building blocks for developers to write their own tools to launch in the environment and leverage the capabilities provided by the environment. The unique capabilities provided help support building products with multiple repositories and having each repository contribute/plugin to the build process in a scalable way. The environment will scan the files in the code tree (multiple repos) and discover plugins, dependencies, path adjustments, environment variable settings, etc. This provides easy methods for common repositories to share build tools/steps.

Inclusion of this package is best managed using Pip/Pypi. This package makes use of edk2-pytool-library.

This is a supplemental package and is not required to be used for edk2 builds.

Current Status

The code coverage and CI badges represent unit test status and the code coverage of those unit tests. We require 100% unit test success (Hence the pass / fail) and that code coverage percentage does not lower.

Host Type	Toolchain	Project	Integration Tests
Windows Server 2019	Python 3.10	Edk2
Windows Server 2019	Python 3.11	Edk2
Windows Server 2019	Python 3.12	Edk2
Linux Ubuntu 20.04	Python 3.10	Edk2
Linux Ubuntu 20.04	Python 3.11	Edk2
Linux Ubuntu 20.04	Python 3.12	Edk2
Windows Server 2022	Python 3.10	Project Mu
Windows Server 2022	Python 3.11	Project Mu
Windows Server 2022	Python 3.12	Project Mu
Linux Ubuntu 22.04	Python 3.10	Project Mu
Linux Ubuntu 22.04	Python 3.11	Project Mu
Linux Ubuntu 22.04	Python 3.12	Project Mu

Current Release

All release information is now tracked with Github tags, releases and milestones.

Content

The package contains cli tools and a basic common environment for running tools. This common environment supports extensions, sub-classing, and plugin to allow great flexibility for building and maintaining a code tree.

Examples:

• CI build support with plugin
• Binary dependency resolution (nuget, urls, git repos)
• Loggers (file, memory, and colored console)
• Plugins (pre/post build, function injection)
• Wrapper around edk2 build
• VarDict and ShellEnvrionment to manage key/value pairs consistently across entire process
• Nuget Publishing tool to push new packages
• Omnicache - Support a super cache of git repos to speed up creating and updating multiple work spaces and minimizing filesystem impact

License

All content in this repository is licensed under BSD-2-Clause Plus Patent License.

Usage

NOTE: It is strongly recommended that you use python virtual environments. Virtual environments avoid changing the global python workspace and causing conflicting dependencies. Virtual environments are lightweight and easy to use. Learn more

• To install run pip install --upgrade edk2-pytool-extensions

• To use in your python code

  from edk2toolext.<module> import <class>

History

This project and functionality was ported from Project Mu. For history and documentation prior to this see the original Project Mu projects https://github.com/microsoft/mu_pip_environment and https://github.com/microsoft/mu_pip_build

Contribution Process

This project welcomes all types of contributions. For issues, bugs, and questions it is best to open a github issue.

Code Contributions

For code contributions this project leverages github pull requests. See github tutorials, help, and documentation for complete descriptions. For best success please follow the below process.

1. Contributor opens an issue describing problem or new desired functionality
2. Contributor forks repository in github
3. Contributor creates branch for work in their fork
4. Contributor makes code changes, writes relevant unit tests, authors documentation and release notes as necessary.
5. Contributor runs tests locally
6. Contributor submits PR to master branch of tianocore/edk2-pytool-extensions
   1. PR reviewers will provide feedback on change. If any modifications are required, contributor will make changes and push updates.
   2. PR automation will run and validate tests pass
   3. If all comments resolved, maintainers approved, and tests pass the PR will be squash merged and closed by the maintainers.

Maintainers

See the github team for more details.

Documentation

Documentation for the most recent release of edk2-pytool-extensions is hosted on tianocore.org/edk2-pytool-extensions. Raw documentation is located in the docs/ folder and is split into two separate categories. The first is located at docs/user/ and is documentation and API references for those that are using this package in their own project. Users can generate a local copy of the documentation by executing the following command from the root of the project:

pip install --upgrade -e .[docs]
mkdocs serve

The second is located at docs/contributor/ and is documentation for contributing to the edk2-pytool-extensions repository.

edk2-pytool-extensions's Issues

Error is green for Platform build

Describe the bug
On platform builds when it should be an error, we emit a progress of "ERROR"

To Reproduce
Steps to reproduce the behavior:

1. Build a platform with an error
2. Note that since it is a progress message, error is green.

Expected behavior
It should be red.

Add spell checking to the PR process

Add spell checking to the CI and PR gate.
Work has been done to add cspell to project mu based projects so it should be easy to add here.

stuart_ci_build should provide a way for a settings file to return options on a per package level

web dependency sha256 value is case sensitive

comparison or sha256 hex string expects lower case.

Make case insensitive and unit test as there is no coverage on the sha256 comparison code

Build.Conf file throw errors if two empty lines are in it

Sample file provided

LaunchBuildLogProgram = "C:\Program Files\Sublime Text 3\subl.exe"  #if you remove this it will launch with notepad by default
LaunchLogOnSuccess = False  #don't want to see build log on success  # set to true if you want to see it
LaunchLogOnError = False  #don't want to see build log on error # set to true if you want to see it
BUILDREPORT_TYPES=PCD DEPEX FLASH BUILD_FLAGS LIBRARY


BUILDREPORTING=TRUE

Throw error when tool_chain_tag is not found when creating conf folder

It can lead to a strange state for our conf folder so it makes sense if we are creating the conf folder that we require tool_chain_tag to be set

NugetDependency has inconsistent behavior surrounding caches and versions.

See PR #105

Need to make the version normalization more consistent, improve test path coverage so all errors are accounted for, and make sure caching is accounted for.

Warnings from non-root loggers are suppressed

BuildConf.config syntax with a trailing comment will not work

See
https://github.com/tianocore/edk2-pytool-extensions/blob/master/edk2toolext/edk2_invocable.py#L158

If the config file has
test=goingToFail #comment here

This will incorrectly try to set test = goingToFail #comment here
whereas it should set test = goingToFail

console logger sends output to stderr not stdout

Add "getting started" usage examples

Looking around the features, I'm really excited to give these tools a try in my own EDK2 projects, but I cannot figure out what steps I need to perform in order to integrate them into my codebase.

docs/using.md tantalizingly states:

You are one settings file away from a great platform building experience.

But I cannot figure out what settings file it is referring to, how to create it, or where to put it.

Basically, it would be really helpful to have a getting started guide that walks through how to take an existing EDK2 project and adapt it to use these tools.

XML log created by Stuart_ci_build has incorrect fields

See python function instead of value.

Linux doesn't generate good error when mono isn't installed

Describe the bug
When mono isn't installed, you get this error:

  File "/home/mattfc/.local/lib/python3.6/site-packages/edk2toolext/environment/self_describing_environment.py", line 205, in update_extdeps
    extdep.fetch()
  File "/home/mattfc/.local/lib/python3.6/site-packages/edk2toolext/environment/extdeptypes/nuget_dependency.py", line 162, in fetch
    if self._fetch_from_cache(package_name):
  File "/home/mattfc/.local/lib/python3.6/site-packages/edk2toolext/environment/extdeptypes/nuget_dependency.py", line 95, in _fetch_from_cache
    if not os.path.isdir(NugetDependency.global_cache_path):
  File "/usr/lib/python3.6/genericpath.py", line 42, in isdir
    st = os.stat(s)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

Which isn't very user friendly.

To Reproduce
Steps to reproduce the behavior:

1. Use WSL/VM/Linux box to get pytools
2. Try to do stuart update
3. See error

Expected behavior
We should advise the user to install mono and suggest where to get it?

Execution environment
Ubuntu 18.04 WSL2 Py3.5

Pip Packages
asn1crypto==0.24.0
attrs==17.4.0
Automat==0.6.0
blinker==1.4
certifi==2018.1.18
chardet==3.0.4
click==6.7
cloud-init==18.5
colorama==0.3.7
command-not-found==0.3
configobj==5.0.6
constantly==15.1.0
cryptography==2.1.4
distro-info===0.18ubuntu0.18.04.1
edk2-pytool-extensions==0.12.0
edk2-pytool-library==0.10.3
httplib2==0.9.2
hyperlink==17.3.1
idna==2.6
incremental==16.10.1
Jinja2==2.10
jsonpatch==1.16
jsonpointer==1.10
jsonschema==2.6.0
keyring==10.6.0
keyrings.alt==3.0
language-selector==0.1
MarkupSafe==1.0
netifaces==0.10.4
oauthlib==2.0.6
PAM==0.4.2
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycrypto==2.6.1
pygobject==3.26.1
PyJWT==1.5.3
pyOpenSSL==17.5.0
pyserial==3.4
python-apt==1.6.4
python-debian==0.1.32
pyxdg==0.25
PyYAML==5.1.2
requests==2.18.4
requests-unixsocket==0.1.5
SecretStorage==2.3.1
service-identity==16.0.0
six==1.11.0
ssh-import-id==5.7
systemd-python==234
Twisted==17.9.0
ufw==0.36
unattended-upgrades==0.1
urllib3==1.22
zope.interface==4.3.2

Additional context
Add any other context about the problem here.

ci_setup doesn't check to make sure dependencies are up to date

It just checks to make sure that they're on the right branch. Maybe commit only? Similar to gitsubmodules?

web dependency doesn't work on single files

when downloading single files (like a binary from github) it fails to create the ext_dep folder

Stuart doesn't give a good error when trying to use the wrong invokable

When doing a stuart_build on a CISettings.py file, it just says 'NoneType' object is not callable - which is fairly cryptic. Checking for this common mistake would cut down on a lot of developer confusion.

Missing information from version report

1. pip versions
2. Python version

stuart_ci_build skipped plugin/tests don't show up as skipped

when plugin is skipped it is not recorded as skipped. No indication that it even exists. This is not desired behavior.

Omnicache path check causes failure when not set

Error when not set.

Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.7.4/x64/bin/stuart_setup", line 10, in
sys.exit(main())
File "/opt/hostedtoolcache/Python/3.7.4/x64/lib/python3.7/site-packages/edk2toolext/invocables/edk2_setup.py", line 219, in main
Edk2PlatformSetup().Invoke()
File "/opt/hostedtoolcache/Python/3.7.4/x64/lib/python3.7/site-packages/edk2toolext/base_abstract_invocable.py", line 105, in Invoke
self.ParseCommandLineOptions()
File "/opt/hostedtoolcache/Python/3.7.4/x64/lib/python3.7/site-packages/edk2toolext/edk2_invocable.py", line 187, in ParseCommandLineOptions
self.RetrieveCommandLineOptions(args)
File "/opt/hostedtoolcache/Python/3.7.4/x64/lib/python3.7/site-packages/edk2toolext/invocables/edk2_setup.py", line 84, in RetrieveCommandLineOptions
if (not os.path.exists(self.omnicache_path)):
File "/opt/hostedtoolcache/Python/3.7.4/x64/lib/python3.7/genericpath.py", line 19, in exists
os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
##[error]/opt/hostedtoolcache/Python/3.7.4/x64/bin/stuart_setup failed with return code: 1

FDF parser does not import build environment from DSCs

We fail to parse FDFs that depend on DEFINES defined in the corresponding DSC

To Reproduce
I'll post HOWTO later

Expected behavior
I can build the KabyLakeRvp, but it breaks

Execution environment
List the OS, python version, system environment variables, etc.

Pip Packages
If python related run python -m pip list and copy the output here

Additional context
DEBUG - Parse Active Platform DSC file: C:\git\edk2-platforms-pytools\Platform\Intel\KabylakeOpenBoardPkg/KabylakeRvp3/OpenBoardPkg.dsc
DEBUG - Getting all build keys for build type DEBUG
DEBUG - Parse Active Flash Definition (FDF) file
DEBUG - Getting all build keys for build type DEBUG
ERROR - Invalid file path C:\git\edk2-platforms-pytools$(PLATFORM_PACKAGE)/Include/Fdf/CorePreMemoryInclude.fdf
CRITICAL - Build Process Exception
ERROR - Traceback (most recent call last):
File "c:\git\venv_edk2-pytools\lib\site-packages\edk2toolext\environment\uefi_build.py", line 109, in Go
ret = self.SetEnv()
File "c:\git\venv_edk2-pytools\lib\site-packages\edk2toolext\environment\uefi_build.py", line 384, in SetEnv
ret = self.ParseFdfFile()
File "c:\git\venv_edk2-pytools\lib\site-packages\edk2toolext\environment\uefi_build.py", line 519, in ParseFdfFile
fdfp.ParseFile(pa)
File "c:\git\venv_edk2-pytools\lib\site-packages\edk2toollib\uefi\edk2\parsers\dsc_parser.py", line 262, in ParseFile
self.__ProcessDefines(file_lines)
File "c:\git\venv_edk2-pytools\lib\site-packages\edk2toollib\uefi\edk2\parsers\dsc_parser.py", line 246, in __ProcessDefines
(line, add) = self.__ParseDefineLine(l)
File "c:\git\venv_edk2-pytools\lib\site-packages\edk2toollib\uefi\edk2\parsers\dsc_parser.py", line 184, in __ParseDefineLine
lf = open(sp, "r")
FileNotFoundError: [Errno 2] No such file or directory: 'C:\git\edk2-platforms-pytools\$(PLATFORM_PACKAGE)/Include/Fdf/CorePreMemoryInclude.fdf'

Stuart_setup doesn't give a good error when unable to clone a specific commit

Stuart_ci_build -p parameter eats all positional parameters following it

Update might not work correctly

One of the keys features we advertise and rely on in stuart update is finding new dependencies if they are inside other dependencies. It looks like Bootstrap uses some global state to save the ENV state, so it won't search the workspace and find the new ext_deps.

Am I wrong here?

def BootstrapEnvironment(workspace, scopes=()):
    global ENVIRONMENT_BOOTSTRAP_COMPLETE, ENV_STATE

    if not ENVIRONMENT_BOOTSTRAP_COMPLETE:
        build_env = self_describing_environment(
            workspace, scopes).load_workspace()

platform build, setup, and update logs should be in the same folder that we are building in

Nuget Publishing should switch from LicenseUrl to License.

May have to have the script fetch the URL contents for backwards compat.

WARNING: NU5125: The 'licenseUrl' element will be deprecated. Consider using the 'license' element instead.

--------------Cmd Output Finished---------------
--------- Running Time (mm:ss): 00:05 ----------

=======================================
Name: Mu-Basetools
ConfigFile: D:\a\1\a/NugetPackageConfig.json
name: Mu-Basetools
author_string: ProjectMu, TianoCore
server_url: https://api.nuget.org/v3/index.json
project_url: https://aka.ms/projectmu
license_url: https://github.com/tianocore/edk2/blob/master/License.txt
description_string: Edk2 Basetools compiled for Windows and Linux; packaged for Project Mu
copyright_string: Copyright 2018 - 2019

git operations standardized for ci_setup and platform_setup

Add tooling to deal with FMP Capsule creation.

location of build log is not displayed on error conditions in build

Nuget dependencies hang when credentials are needed

Git Dependencies causes error with version aggregator

Describe the bug
When cloning a git dependency for the first time, we report the version twice. Once before we clone and once after. Since the path before the clone is none, the paths don't match and it throws an error.

To Reproduce
Steps to reproduce the behavior:

1. Clone a git ext dep in your tree via stuart_setup
2. See error

Additional context
Add any other context about the problem here.

stuart_ci_setup doesn't give much information

You don't get much output or progress other than that it is doing something. "Cloning/check repo X" would be helpful

uefi_build assumes MAX_CONCURRENT_THREAD_NUMBER is set

MAX_CONCURRENT_THREAD_NUMBER is not required in Target.txt therefore uefi_build should support case when that variable isn't set. In fact default tianocore target.txt does not define this value.

Credential provider is needed for submodules

Remove manual structure packing from capsule_helper

Currently, there is some manual structure packing in build_capsule(). Change this to use a method of the appropriate class to retrieve the "To Be Signed" data.

Version Aggregator is constantly warning about duplicate entries

stuart_update logging needs cleaned up

Example log looks like:

SECTION - Init SDE
SECTION - Loading Plugins
SECTION - Start Invocable Tool
SECTION - First Update
SECTION - Retry Count: 1
SECTION - Summary
PROGRESS - Success

I would suggest First Update and Retry Count lines are not good.

A pattern of doing Update and then if during the update a new ext_dep was found the next message can indicate that another update is needed due to environmental changes.

ConfMgmt fails

When the conf specifies a invalid file to build with, it shows a generic error

INFO - The system cannot find the path specified.
INFO -
INFO -
INFO - Build...
INFO - : error 7000: Failed to execute command
INFO - ....\Vc\Bin\nmake.exe /nologo tbuild [c:\git\uefi2018\Build\MsftIvanhoePkg\DEBUG_VS2015x86\IA32\MdePkg\Library\PeiPcdLib\PeiPcdLib]
INFO -

web dependency test coverage is lacking for folders and compression

Web dependency test coverage shows the compression code is not covered

Remove GetModulePkgsPath from Platform Build

Use GetPackagesPath and then translate it to ";".join in go

Stuart_CI_BUILD doesn't show the error spew when it fails

This might be more related to the compiler plugin

If there's an error in the settings file, it don't show you what it is

Error: We had trouble loading NetworkPkg\SharedNetworking\SharedNetworkSettings.py. Is the path correct?
usage: DriverBuilder.py [-c PLATFORM_MODULE]

optional arguments:
-c PLATFORM_MODULE, --platform_module PLATFORM_MODULE
Provide the Platform Module relative to the current
working directory.This should contain a
BinaryBuildSettingsManager instance.

Setup, Update, and Build SettingsManager can't optimize based on user supplied CLI options because there is no sharing of that back to settings manager

In trying to optimize the ci build time more has been identified that can be done if the SettingsManagers know more about what is going to actually build. This means things like individual packages, architectures, and targets. The SettingsManagers must be notified of the CLI settings prior to providing the active scopes or required repos.

A few optimizations that should be supported:

• If a run is not going to build cryptopkg then it might not need the OpenSSL submodule.
  • This means that stuart setup, ci setup, and update needs to optionally be told what package is going to be built so that submodules and dependencies could be managed.
• A run might only want to compile for DEBUG or RELEASE.
  • The Target should be settable on the cli.
• A run might only want to compile/test for a given arch
  • Arch should be settable on the cli

Add docs for new edk2_pr_eval tool

GetPackagesPath should map paths to be workspace relative if they aren't already

edk2_pr_eval policy 3 is not first checking that change is in public file

The edk2 pr eval tool checks the contents of a PR and attempts to evaluate the impact on all the packages of the code tree. For "Policy 3" it says it detects if the changed file is a public file of a package and if so it finds all packages dependent on that package and marks them as needing to be tested.
Unfortunately in version 0.11.3 the check is not first checking if it is a public file and therefore even changes made in internal files (like libraries or modules) of a package are triggering dependent packages to be tested. This is in turn causing the PR build to test too much.

Configuration population fails in strange ways when environmental variables aren't set

ERROR - Traceback (most recent call last):
File "c:\git\edk2-pytool-extensions\edk2toolext\environment\uefi_build.py", line 109, in Go
ret = self.SetEnv()
File "c:\git\edk2-pytool-extensions\edk2toolext\environment\uefi_build.py", line 369, in SetEnv
conf_mgmt.ConfMgmt().populate_conf_dir(conf_dir, self.UpdateConf, TemplateDirList)
File "c:\git\edk2-pytool-extensions\edk2toolext\environment\conf_mgmt.py", line 58, in populate_conf_dir
p = os.path.join(r, templatefiles[x])
File "c:\python\3.7.2\lib\ntpath.py", line 76, in join
path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

stuart_ci_build should not put all dependency in the PackagesPath

Move "version normalization" code from capsule_helper to edk2toollib inf_generator.

This code makes more sense living alongside the generator class itself.

When a build variable is updated with same value but different attributes those attributes are not set

If a build variable is set to value1 and allow override. On a future set where the value is still value1 but override is now false the vardict is not updated to include the new state of override.