tianocore / edk2-edkrepo

Repository for the edkrepo utility

EdkRepo - The Multi-Repository Tool for EDK II

Introduction

EdkRepo is the multi-repository tool for EDK II firmware development. EdkRepo is built on top of git. It is intended to automate common developer workflows for projects that use more than one git repository. For example many of the new projects in the edk2-platforms repository require the user to clone several git repositories. EdkRepo makes it easier to set up and upstream changes for these projects. EdkRepo does not replace git, rather it provides higher level extensions that make it easier to work with git. EdkRepo is written in Python and is compatible with Python 3.5 or later.

Linux Build and Installation

Install Pre-Requisites

• Git 2.13.x or later
• Python 3.5 or later
• Python SetupTools
• Python Pip

Ubuntu Specific Instructions

Tested versions: 22.04 LTS, 20.04 LTS, 18.04 LTS, 16.04 LTS sudo apt-get install git python3 python3-setuptools python3-pip

Upgrade git (Ubuntu 16.04 LTS Only)

The version of git that is installed by default in Ubuntu 16.04 is too old for EdkRepo (16.04 includes git 2.7.4, the minimum is 2.13+). To upgrade git, run the following commands:

sudo apt-add-repository ppa:git-core/ppa

Press [ENTER] to confirm that you want to add the new repository.

sudo apt-get update

sudo apt-get install git

OpenSUSE Specific Instructions

sudo zipper install git python3 python3-setuptools python3-pip

Red Hat/Fedora/Rocky Specific Instructions

sudo dnf install git python3-pip

Install EdkRepo

Installing EdkRepo on Linux requires one to extract the tarball and run the included installer script.

1. Extract the archive using the following command tar -xzvf edkrepo-<version>.tar.gz
2. Run the installer script ./install.py and follow the on-screen prompts.

The -v flag can be added for more verbose output if desired.

Automated Installation

For an automated non-interactive install, one must provide at least 2 arguments: --local or --system and --prompt or --no-prompt.

--local requests installation to the current user's home directory. Root access is not required and no system wide changes are made. This is the recommended installation method.

--system request a system-wide installation. Root access is required. This method is useful for systems where multiple users will be running EdkRepo as it saves disk space in that scenario.

For system level installations, one must provide the --user parameter and run the installation script as root.

Build Process

To build a EdkRepo distribution tarball, the Python wheel package is required in addition to the above dependencies. On Ubuntu, one can install it using:

sudo apt-get install python3-wheel

1. cd build-scripts
2. ./build_linux_installer.py

Install From Source

To install from source, one must have installed using the tarball method above at least once in order to setup the EdkRepo configuration files. One this is done, one may use the standard distutils method to install EdkRepo from source:

./setup.py install

macOS Build and Installation

Install Pre-Requisites

1. Install the Xcode Command Line Tools

a) Open a Terminal and type the following command:

xcode-select --install

b) A new window will appear, click Install. c) Accept the license agreement. d) Wait for the installation to complete.

2. Install Homebrew

Install Homebrew if it has not been installed already. Homebrew is a package manager for macOS that has become the most common method of installing command line software on macOS that was not originally provided by Apple. EdkRepo has several dependencies that are distributed via Homebrew.

Type the following command to install Homebrew:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Follow the on-screen prompts.

3. Install Dependencies

Run the following commands to install EdkRepo's dependencies:

brew install bash-completion git git-gui xz pyenv

pyenv install 3.8.16

pyenv global 3.8.16

During installation, you may be prompted to enter your password.

4. Configure Shell for Pyenv and Git

To enable usage of Pyenv installed Python interpreters and Git command completions, run the following command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/tianocore/edk2-edkrepo/main/edkrepo_installer/mac-scripts/setup_git_pyenv_mac.sh)"

Restart your shell so the Pyenv changes can take effect:

exec $SHELL

Install EdkRepo

Extract the archive:

tar -xzvf edkrepo-<version>.tar.gz

If you are installing from source, you will need to build the distribution tarball using the following commands first:

1. pip install wheel (If not done already)
2. cd build-scripts
3. ./build_linux_installer.py

Install EdkRepo:

./install.py

Restart your shell so the new Pyenv shim for EdkRepo can take effect:

exec $SHELL

Windows Build and Installation

Pre-Requisites

• Git 2.13.x or later
• Python 3.8.8 or later

Git 2.27.0 is the version that has received the most validation, though any version of Git 2.13 or later works fine. If you want to install 2.27.0, here are some links:

• Direct Link - Git for Windows 2.27.0 - 64 Bit
• Direct Link - Git for Windows 2.27.0 - 32 Bit

Python 3.8.8 or later is recommended due to performance improvements and CVE-2021-3177. You can get Python from here: https://www.python.org/
Windows installer .exe will fail if Python 3.8.8 or later is not detected.

Install Process

1. Run the installer .exe
2. Click Install

Install From Source

To install from source, one must build and run the installer .exe using the instructions below at least once in order to setup the EdkRepo configuration files. One this is done, one may use the standard distutils method to install EdkRepo from source:

py -3 setup.py install

Build Process

Build Pre-Requisites

• Visual Studio 2015 or later with the C# language and C++ compiler installed
• Python Wheel

Install Python wheel using the following:

py -3 -m pip install wheel

Open a command prompt and type the following:

1. cd build-scripts
2. build_windows_installer.bat

Timeline

Time	Event
WW 10 2021	Moved from edk2-staging to a dedicated repository
WW 26 2019	Initial commit of EdkRepo
...	...

Maintainers

• Ashley DeSimone [email protected]
• Nate DeSimone [email protected]

Known Issues

Please see https://github.com/tianocore/edk2-edkrepo/issues

Related Links

• https://github.com/tianocore/edk2-edkrepo-manifest
• https://github.com/tianocore/edk2-platforms

edk2-edkrepo's Issues

Update Error Codes in edkrepo_exception.py

The error codes defined in edkrepo_exception.py should be consecutive starting at 100 ex the following:

class EdkrepoWorkspaceCorruptException(EdkrepoException):
    def __ini__(self, message):
        super().__init__(message, 118)

class EdkrepoWarningException(EdkrepoException):
    def __init__(self, message):
        super().__init__(message, 123)

class EdkrepoGitException(EdkrepoException):
    def __init__(self, message):
        super().__init__(message, 129)

Should change to:

class EdkrepoWorkspaceCorruptException(EdkrepoException):
    def __ini__(self, message):
        super().__init__(message, 118)

class EdkrepoWarningException(EdkrepoException):
    def __init__(self, message):
        super().__init__(message, 119)

class EdkrepoGitException(EdkrepoException):
    def __init__(self, message):
        super().__init__(message, 120)

Checkout pins is failing due to changed function parameters

checkout_repos() added one parameter: global_manifest_path

Manifest Parser Must Support Listing A PatchSet in a repo source

The manifest parser must support listing a PatchSet in the section of Combinations and return the name of the patch set in the repo source tuple.

<Combination> <Source localRoot=" remote='' (branch|commit|tag) or patchSet='' /> </Combination>

The support for branch/commit/tags and patchSets must be mutually exclusive; if both are present then parsing must fail.

This issue depends on the completion of #73

[Enhancement] extend edkrepo log command line argument

Brief: Introduce more kwargs option in "edkrepo log" command

Command "edkrepo log" is defined under:

• edk2-edkrepo/edkrepo/commands/log_command.py

It retrieve commit messages through gitpython module called under:

• edk2-edkrepo/edkrepo/commands/common_repo_functions.py -> sort_commits()

The API being used from gitpython is Repo.iter_commits()

• from implementation, sort_commits feeds either empty argument or max_count to iter_commits
• The API "iter_commits" doc "https://gitpython.readthedocs.io/en/stable/reference.html" indicate it can accept all argument from git-rev-list

Hence iter_commits could accept more argument such as:

• iter_commits("origin/master..HEAD")
• majority of the git log arguments

Feature request:

• Adding kwargs to iter_commits to extend "edkrepo log" command line option
  Result:
• "edkrepo log" can be more "git log" like command

Error: descriptor 'casefold' for 'str' objects doesn't apply to a 'NoneType' object

EdkRepo clone must create an appropriate Python virtual environment.

If combo being cloned has venv_enable == True defined in the project manifest file EdkRepo must create and configure an appropriate python virtual environment.

• The virtual environment must be named based on the combo
• all configuration files for the virtual environment must be stored in the workspace/repo directory
• all pip requirements listed in the repo sources for the selected combo must be installed into the virtual environment
• if conflicting requirements are found the user must be warned and no virtual environment will be created

Update f2f-cherry-pick to use new standardized message formatting

All prints in the f2f-cherry-pick command should be updated to use the new print methods defined in common/ui_functions.py

edkrepo clean -x request

Can 'edkrepo clean' add support for [-x] as an argument similar to 'git clean [-x]' ?

e.g. 'edkrepo clean -fdx'

With per combo sparse checkout settings the edkrepo combo is not performning the sparse checkout correctly.

With per combo sparse checkout settings the edkrepo combo is not performning the sparse checkout correctly.

edkrepo checkout X

has different results than

edkrepo sparse --disable

edkrepo sparse --enable

Manifest Parser Must Support Defining Patch Sets

The manifest parser must support defining PatchSets with the following syntax. These patch sets can be referenced in the source's defined in a combo and be leveraged by EdkRepo dynamically create custom local branches for users based on a recipe.

This ticket is to add initial support for parsing these new sections.

PatchSets

<PatchSets>
    <PatchSet remote='' name='' parentSha='' fetchBranch='' />
        <Patch file=''/>
        <Revert sha=''/>
        <CherryPick sha='' sourceRemote='' sourceBranch=''/>
    </PatchSet>
</PatchSets>

• An arbitrary number or patch sets may be contained within the PatchSets section
• parentSha may refer to a sha or another PatchSet
• Patch:  refers to a global manifest repo relative path to a .patch file that can be applied via git am
• Revert: refers to a SHA from the given repo/branch to be reverted using git revert
• CherryPick: refers to a SHA to be cherry picked onto the newly created local branch using git cherry-pick
• sourceRemote: an optional field denoting an alternate repository to cherry pick from. During the process of creating the local branch this remote will be temporarily added to the target remote; If sourceBranch is not defined when using sourceRemote then a git fetch --all will be done to ensure that the SHA is available. 
• sourceBranch: an optional field denoting which branch to cherry pick the SHA from. If no sourceRemote is defined it will be assumed that this branch exists within the target remote
• If neither sourceRemote or sourceBranch are defined EdkRepo will assume that the SHA exists within the target remote and will run git fetch --all to ensure that the SHA is present on the users system
• remote: Required field denoting the target remote as defined in the RemoteList section of the manifest
• name: The name of the patch set will be converted to all lower case during parsing; this name also denotes the name of the local branch to be created by EdkRepo
• parentSha:  The starting point for the newly created local branch. To be used by git branch / git checkout -b; this may also reference an additional patchset as defined in the manifest file as a starting point.
• fetchBranch: The name of the branch to fetch in order for the parentSha to be present on the users system.

Usage of PatchSet Data

PatchSet data should be made available as a named tuple following the precedent set by other publicly accessible data from the parser and should be as follows:
PatchSet = namedtuple('PatchSet', ['remote', 'name', 'parent_sha', 'fetch_branch'])

Individual patch operations (patch, revert and cherry-pick) should also be implemented as named tuples defined as follows:
PatchOperation = namedtuple('PatchOperation',['type', 'file', 'sha', 'source_remote', 'source_branch'])
Type can be one of: Patch, Revert or CherryPick. Data not used by the given operation type should be populated as None

The following methods and/or properties should be implemented:

• get_all_patchsets():returns a list of PatchSet tuples enumerating all PatchSets defined in the manifest file
• get_patchset_operations(name): Takes the name of a patchset and returns list of PatchOperations preserving the order in which they were entered in the manifest file.

When a branch tracking json is deleted throw an error

List Pins Command Fails

When run without the -v flag no output is given.

When the -v flag is provided the following stack trace is output:
Traceback (most recent call last): File "C:\Python38\lib\site-packages\edkrepo_internal\edkrepo_cli.py", line 304, in main command.run_command(command_name, parsed_args, config) File "C:\Python38\lib\site-packages\edkrepo\commands\composite_command.py", line 38, in run_command return command.run_command(args, config) File "C:\Python38\lib\site-packages\edkrepo\commands\list_pins_command.py", line 125, in run_command less_output = subprocess.Popen([str(less_path), '-F', '-R', '-S', '-X', '-K'], stdin=subprocess.PIP E, stdout=sys.stdout, universal_newlines=True) File "C:\Python38\lib\subprocess.py", line 808, in __init__ errread, errwrite) = self._get_handles(stdin, stdout, stderr) File "C:\Python38\lib\subprocess.py", line 1178, in _get_handles c2pwrite = msvcrt.get_osfhandle(stdout.fileno()) io.UnsupportedOperation: fileno

edkrepo can create Dynamic Branch from a PatchSet parentSha set to a Branch name

Create a PatchSet or two that reference a git branch name as the parentSha.

e.g. the "master" branch of the Tianocore Edk2 repo:

<PatchSet remote="Edk2Repo" name="edk2_test_B" parentSha="edk2_test_A" fetchBranch="master">
</PatchSet>
<PatchSet remote="Edk2Repo" name="edk2_test_A" parentSha="master" fetchBranch="master">
</PatchSet>

When using these PatchSet as part of a combination, if there is not a 'tag' on the repo that conflicts with the 'branch' name, then the PatchSet will use the branch as the base of the Dynamic Branch Creation PatchSet branch.

Expected result:
PatchSet were proposed to report an error and revert to previous combo if the base parentSha is not a 'tag' or a commit 'SHA', since no way to guarantee a PatchSet can apply without a merge error to a changing Branch codebase.

Test system: edkrepo 3.1.0, python 3.8.8, Windows 11

EdkRepo Sync Should Still Work When the Project Manifest Cannot be Located in a Manifest Repo

When the lookup of a project fails the sync command should alert the user that they cannot located the entry in a manifest repository but proceed with syncing the workspace.

checkout between similar Combination can leave stale "HEAD detached at" status.

"edkrepo checkout" between two combo with similar tags does not update the "HEAD detached at " message for the repo.

Create three tags on a repo, tag_a , tag_b, tag_a2, where tag_a and tag_a2 are associated to the same commit.
Create three Combination in a manifest, combo_a, combo_b, and combo_a2 that use the tags above.

Checkout combo_a , then combo_a2, and check status:

edkrepo checkout combo_a
edkrepo checkout combo_a2
edkrepo status 

edkrepo status (and git status) shows "HEAD detached at tag_a", functionally correct, but unexpected for combo_a2.

Checkout combo_a , combo_b, then combo_a2, and check status:

edkrepo checkout combo_a
edkrepo checkout combo_b
edkrepo checkout combo_a2
edkrepo status 

edkrepo status (and git status) shows "HEAD detached at tag_a2" as expect for combo_a2.

Similar behavior when running just git commands below:

git checkout tag_a
git checkout tag_a2
git status

Tested on git version 2.39.1.windows.1

Installing EdkRepo causes X11 to crash at startup on Raspberry Pi OS

Installing EdkRepo on Raspberry Pi OS will cause X11 to crash during startup on the next reboot. This causes the desktop environment to become inaccessible.

The root cause for this issue is that the X11 startup script on Raspberry Pi will blindly run all scripts in /etc/profile.d using /bin/sh (which defaults to dash on Raspberry Pi) without any regard for the #! line at the start of the script. This seems like a bug in Raspberry Pi OS.

Attempting to execute the EdkRepo bash completions using dash will result in syntax errors. The reason for this is setting command completions requires one to use the COMREPLY variable, which is required to be an array. Arrays are a bash-specific extension that is not in the POSIX standard, so that syntax will be considered invalid.

Raspberry Pi OS installs a script in /etc/profile.d that will check if the current interpreter is bash and then run the command completion scripts in /usr/share/bash-completion/completions. Therefore, the workaround for this issue is to place the command completion script there. This also has the benefit of bash-completions providing a performance boost by only running completion scripts for the command that is currently being typed by the user.

checkout combo with PatchSet failed does not revert to previous combo

checkout combo with PatchSet failure does not revert to previous combo

Steps to reproduce:

Clone the Intel-MinPlatform edkrepo project.
Modify the /repo/Manifest.xml file to add a new local combination with an invalid PatchSet.
Example below reverts the same commit twice in a row.

<PatchSets>
  <PatchSet remote="Edk2Repo" name="edk2_negative_test" parentSha="edk2-stable202302" fetchBranch="master">
    <Revert sha="f80f052277c88a67c55e107b550f504eeea947d3" />
    <Revert sha="f80f052277c88a67c55e107b550f504eeea947d3" />
  </PatchSet>
</PatchSets>


<Combination name="negative_test" description="negative patchset test">
  <Source localRoot="edk2" remote="Edk2Repo" patchSet="edk2_negative_test" sparseCheckout="true" enableSubmodule="true" />
  <Source localRoot="edk2-platforms" remote="Edk2PlatformsRepo" branch="master" sparseCheckout="true" />
  <Source localRoot="edk2-non-osi" remote="Edk2NonOsiRepo" branch="master" sparseCheckout="true" />
  <Source localRoot="FSP" remote="FspRepo" branch="master" sparseCheckout="false" />
</Combination>

Attempt to checkout the new local "negative_test" combo. The PatchSet failure is detected, but the combo continues to be checked out.

"""
Failed to revert to the commit f80f052277c88a67c55e107b550f504eeea947d3
Failed to apply one of the patchset operations. Checking out back to the default branch
Checking out master branch for edk2-platforms repo ...
Checking out master branch for edk2-non-osi repo ...
Checking out master branch for FSP repo ...
"""

Expected result: a detected bad PatchSet failure should be treated as a checkout combo failure, and trigger a revert to the previous checkout combo, or to the default combo if no previous combo.

test system:
edkrepo 3.1.0, Python 3.8.10, git version 2.39.1.windows.1, Windows 11

Edkrepo fails to install with Python3.11

Accidentally, I installed Python 3.11.
Windows set the Python launcher to 3.11.

I tried installing edkrepo, and it failed.

I am only opening this issue as FYI. There may be some dependencies that will break the edkrepo installation in the future.

EdkRepo must provide support to enumerate all patch sets listed in a manifest and create local branches from them

Must be completed after #75 #77 #76

EdkRepo must provide a set of common functionality to do the following:

• Determine the list of patch sets consumed by the combos defined in the manifest file
• For each patch set create a local branch by applying the patch set operations in the order they were listed in the manifest
• If a patch set requires the use an an alternate remote for the purpose of cherry picking EdkRepo must temporarily add that remote and then remove it after the creation of the local branch.
• if any patch set operation fails to apply an exception should be thrown.
• The method for creating the local branch must take the branch name as an input.

Dynamic Branch Creation: Pin Files Should Contain PatchSets Section

The generate_pin_etree() method should also include the complete PatchSets section in the pin file.

Additionally the following section should not be based solely on a tag, branch or SHA:

for src_tuple in repo_source_list: if (src_tuple.root is None or src_tuple.remote_name is None or src_tuple.remote_url is None or (src_tuple.commit is None and src_tuple.branch is None and src_tuple.tag is None)): raise ValueError("Invalid input: empty values in source list")

The logic should be:

if (src_tuple.root is None or src_tuple.remote_name is None or src_tuple.remote_url is None or ((src_tuple.commit is None and src_tuple.branch is None and src_tuple.tag is None) and src_tuple.patch_set is None)):

It should also be verified that the src_tuple lists either a (branch, tag and/or SHA) or a PatchSet

Error: The GitHub username was not found.

Maintenance Command Does Not Clean 'Include Ifs' from GitConfig

Currently the maintenance command only prints out that it is removing stale include ifs from the git config file.

Pin file names do not resolve properly

When using just the name of a pin file it is unable to be located in the global manifest repo. In this case .xml must be appended to the name to properly locate the file.

Add an optional MinimumEdkrepoVersion field for manifest files

If edkrepo is run on manifest files that were designed for newer edkrepo tool versions, the commands could fail entirely, or known bugs could be encountered.

There is not yet a way for a manifest maintainer to notify the manifest consumers about which edkrepo tool version is required to use the latest manifest file version, or notify the manifest consumer if the version will change (e.g. if a newer Manifest update will included a combo with a multi-patch daisy-chained PatchSet, I want the manifest file to enforce a minimum edkrepo tool version of 3.1.1)

I request an optional MinimumEdkrepoVersion field in the ProjectInfo section to be able to specify the minimum EdkRepo tool version required to use the current latest Manifest file.

If a user tries 'edkrepo clone', but the manifest MinimumEdkrepoVersion is not met, the clone should notify the user of the minimum EdkRepo version required, and prevent the edkrepo clone.

If a user tried 'edkrepo sync -u', but the new manifest will have a higher MinimumEdkrepoVersion than the local user's EdkRepo tool, then the user should be notified that a new minimum EdkRepo version is required and the local manifest file will not be updated.

Add option for JSON Output to list-repos

Add the ability to get the list of repositories that EdkRepo knows about in either human readable text or in machine parseable JSON format.

Enable Linux build script to be run on Windows systems

Title

[Bug] Unicode character \u200b (Zero width space) encountered while running f2f-cherry-pick

There is a unicode character \u200b (Zero width space) because of which there are errors in running f2f-cherry-pick. The other commands are running fine. So, the issue is with some file related to f2f-cherry-pick.

unable to install EdkRepo

Getting the following error during install:

Found Python 3.11.1
Git Version: 2.27.0
Running git lfs install
Searching for existing EdkRepo installations...
Determining currently installed Python packages...
Uninstalling current version of edkrepo
Version 4.0.0 of package smmap is already installed
Version 4.0.7 of package gitdb is already installed
Version 3.1.14 of package GitPython is already installed
Version 0.4.4 of package colorama is already installed
Installing edkrepo version 3.1.0
Windows directory is in %PATH%.
Creating edkrepo launcher in Git Bash...
Creating python launcher in Git Bash...
Creating python3 launcher in Git Bash...
Installing EdkRepo command completion...
Copied win_edkrepo_prompt.sh
C:\windows\edkrepo.exe
C:\Program Files\Git\etc\profile.d\edkrepo_completions.sh
Traceback (most recent call last):
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\edkrepo_entry_point.py", line 94, in main
return(func())
^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\edkrepo_cli.py", line 161, in main
command = command_factory.create_composite_command()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 85, in create_composite_command
commands = get_commands()
^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 73, in get_commands
if in_same_module and _is_command(cls[1]):
^^^^^^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 32, in _is_command
if func[0] == 'get_metadata' and len(inspect.getargspec(func[1])[0]) == 1:
^^^^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'getargspec'
Traceback (most recent call last):
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\edkrepo_entry_point.py", line 94, in main
return(func())
^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\edkrepo_cli.py", line 161, in main
command = command_factory.create_composite_command()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 85, in create_composite_command

Only flag a repo source as different when key identifiers do not match

In the sync -u flow the repo source tuples for the initial and new sources are compared. Items that are acceptable to change such as branch, sparse checkout status, etc... are used in this calculation causing the source to be erroneously identified as different and raising a manifest changed exception

3.1.0 release Windows installer flagged as unrecognized app

The Edkrepo 3.1.0 release Windows installers is flagged by Windows 11 as a risky app.

Steps to reproduce:

Download the EdkRepoSetup-3.1.0.0.exe Windows installer from edk2-edkrepo releases page.

https://github.com/tianocore/edk2-edkrepo/releases/download/edkrepo-v3.1.0/EdkRepoSetup-3.1.0.0.exe

Attempt to run EdkRepoSetup-3.1.0.0.exe from Downloads folder.

Windows protected your PC

Microsoft Defender SmartScreen prevented an unrecognized app from starting. Running this app might put your PC at risk.
More Info

App: EdkRepoSetup-3.1.0.0.exe
Publisher: Unknown publisher

Workaround:
Click "More info", then click "Run anyway".

Test system: Windows 11

Request the next edkrepo release provide a Windows installer that won't trigger this warning for new users.

[Customer Find] EDK Repo Setup Install Fails on Windows with Python 3.11.1

Found Python 3.11.1
Git Version: 2.39.1
Running git lfs install
Searching for existing EdkRepo installations...
Determining currently installed Python packages...
Uninstalling current version of edkrepo
Version 4.0.0 of package smmap is already installed
Version 4.0.7 of package gitdb is already installed
Version 3.1.14 of package GitPython is already installed
Version 0.4.4 of package colorama is already installed
Installing edkrepo version 3.1.0
Windows directory is in %PATH%.
Creating edkrepo launcher in Git Bash...
Creating python launcher in Git Bash...
Creating python3 launcher in Git Bash...
Installing EdkRepo command completion...
Copied win_edkrepo_prompt.sh
C:\windows\edkrepo.exe
C:\Program Files\Git\etc\profile.d\edkrepo_completions.sh
Traceback (most recent call last):
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\edkrepo_entry_point.py", line 94, in main
return(func())
^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\edkrepo_cli.py", line 161, in main
command = command_factory.create_composite_command()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 85, in create_composite_command
commands = get_commands()
^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 73, in get_commands
if in_same_module and _is_command(cls[1]):
^^^^^^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 32, in _is_command
if func[0] == 'get_metadata' and len(inspect.getargspec(func[1])[0]) == 1:
^^^^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'getargspec'
Traceback (most recent call last):
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\edkrepo_entry_point.py", line 94, in main
return(func())
^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\edkrepo_cli.py", line 161, in main
command = command_factory.create_composite_command()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 85, in create_composite_command
commands = get_commands()
^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 73, in get_commands
if in_same_module and _is_command(cls[1]):
^^^^^^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 32, in _is_command
if func[0] == 'get_metadata' and len(inspect.getargspec(func[1])[0]) == 1:
^^^^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'getargspec'. Did you mean: 'getargs'?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\windows\edkrepo.exe_main.py", line 7, in
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\edkrepo_entry_point.py", line 99, in main
return edkrepo.edkrepo_cli.main()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\edkrepo_cli.py", line 161, in main
command = command_factory.create_composite_command()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 85, in create_composite_command
commands = get_commands()
^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 73, in get_commands
if in_same_module and _is_command(cls[1]):
^^^^^^^^^^^^^^^^^^^
File "C:\Users\klwoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\edkrepo\commands\command_factory.py", line 32, in _is_command
if func[0] == 'get_metadata' and len(inspect.getargspec(func[1])[0]) == 1:
^^^^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'getargspec'. Did you mean: 'getargs'?
Unable to launch preferred entry point. Launching default entry point edkrepo.edkrepo_cli.py
Installation Error:
System.InvalidOperationException: generate-command-completion-script failed with status 1
at TianoCore.EdkRepoInstaller.InstallWorker.PerformInstall(Action2 ReportComplete, Action1 ReportProgress, Action1 AllowCancel, Func1 CancelPending)
at TianoCore.EdkRepoInstaller.ProgressWindow.WorkerThread()
Installation Failed.

Manifest Parser must provide functionality to get the patchsets referenced in a given combo.

The manifest parser must provide a method get_patchsets_for_combo(combo_name) which returns a dictionary with the key being the source.name for the Source listing the patchset and the value being a PatchSet Tuple

dynamic branch json file corruption during edkrepo sync

During an "edkrepo sync --update-local-manifest" call, the dynamic branch creation json file can be corrupted by the addition of a new branch name row of the currently checked-out dynamic branch being added directly under the first "patch_file" instance in the json file.

example:

"patch_file": [],
"edk2_test1": "edk2_test1_2023/01/27_15_53_34"

This json file corruption leads to two new branches being created during the sync, with the messages:
"Info: A branch with the same name detected. Renaming the old..."
"Info: A branch with the same name detected. Renaming the old..."

After the json file is corrupted, additional "edkrepo checkout [combo]" result in each displaying the message "Info: A branch with the same name detected. Renaming the old..."
Then creates another new dynamic branch, and modifies again the name of the extra branch located under the first "patch_file" instance.

Seen on edkrepo 3.1.0, Python 3.8.10, Windows 11

edkrepo clone [test manifest]
//json file is created with default dynamic branch
edkrepo checkout [test combo]
//json file adds non-default dynamic branch
edkrepo sync --update-local-manifest
//json file corruption occurs

Move Sync Command Specific Strings out of the Common Strings File

For better organization sync command specific strings should be removed from the common string location

Add "edkrepo clean" support for "-ff" option.

Add "edkrepo clean" support for "-ff" option.

"git clean" supports "-f" and "-ff".
https://git-scm.com/docs/git-clean
"Git will refuse to modify untracked nested git repositories (directories with a .git subdirectory) unless a second -f is given."

With edkrepo 3.1.1, "edkrepo clean" does not error when "-ff" is used, but only supports the behavior of "-f".

example edk2 test setup:

Create a Combination that uses enableSubmodule on newer edk2 stable tag.

<Source localRoot="Edk2" remote="Edk2Repo" tag="edk2-stable202305" enableSubmodule="true" />

Manually checkout older Edk2 stable tag.

git checkout edk2-stable202105
git status

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        MdePkg/Library/BaseFdtLib/
        MdePkg/Library/MipiSysTLib/
        UnitTestFrameworkPkg/Library/GoogleTestLib/
        UnitTestFrameworkPkg/Library/SubhookLib/

"edkrepo clean -ffd" runs, but does not remove Untracked files.

"git clean -ffd" runs, and removes Untracked files.

Message informing user that the sync -o flag is required is missing unless -v is given

when running edkrepo sync -u and there is a change in the cloned repository list, the most recent version of EdkRepo prints:

Failed to update manifest.

Without any additional information. When running edkrepo sync -uv you get the following additional message:

The latest manifest for project, <project_name>, requires a change in currently cloned repositories.
To complete this operation please rerun the command with the --override flag
Exiting without performing sync operations.
Failed to update manifest.

This message should not be gated by --verbose

Only clone the specified branch of the global manifest repo.

The definition of manifest repos whether in the edkrepo.cfg or edkrepo_user.cfg file include the repository URL, local directory to be cloned to and the branch to be used as shown below:

[manifest-repos]
edk2-edkrepo-manifest

[edk2-edkrepo-manifest]
URL = https://github.com/tianocore/edk2-edkrepo-manifest.git
Branch = main
LocalPath = edk2-edkrepo-manifest-main 

To improve performance and reduce disk usage the clone of the global manifest repository should only include the specified branch. For the above configuration file this could be accomplished using the following git clone command see https://git-scm.com/docs/git-clone:

git clone https://github.com/tianocore/edk2-edkrepo-manifest.git edk2-edkrepo-manifest-main --single-branch --branch main

The method pull_single_manifest_repo() in edk2-edkrepo\edkrepo\common\manifest_repos_maintenance.py should be updated to use the --single-branch and --branch flags to accommodate this. 

edkrepo is not functioning with Windows install from source steps

On Windows 11 with Python 3.7.9 installed, followed the build steps in the readme.md

py -3 -m pip install wheel
cd build-scripts
build_windows_installer.bat

cd ..
py -3 setup.py install

edkrepo commands fail with the error:
The system cannot find the file C:\windows\edkrepo.exe.

Running the generated "dist/EdkRepoSetup-3.1.0.0.exe" file fails with:
"""
Found Python 3.7.9
Error: Python version 3.8.8 or later is required to run EdkRepo. Please install it before continuing.
Installation Failed.
"""

If edkrepo is going to say it has Python 3.5 and later support on Windows, need one of the install methods such as the install from source steps to work for Python 3.7.9.

Administrator can cause "fatal: detected dubious ownership in repository" errors

On Windows 11, in the "C:\ProgramData\edkrepo" folder, delete the "edk2-edkrepo-manifest-main" manifest-repo folder if it has already been cloned before.

Run a Command Prompt as Administrator, and run "edkrepo manifest". A new "Cloning global manifest repository" will occur.
"Cloning global manifest repository to: C:\ProgramData\edkrepo\edk2-edkrepo-manifest-main from: https://github.com/tianocore/edk2-edkrepo-manifest.git"

Run a Command Prompt as a non-Administrator, and run "edkrepo manifest". "edkrepo manifest" command fails with the following error:

"""
The git command: git remote get-url --all origin failed to complete successfully with the following errors.

stderr: 'fatal: detected dubious ownership in repository at 'C:/ProgramData/edkrepo/edk2-edkrepo-manifest-main'
'C:/ProgramData/edkrepo/edk2-edkrepo-manifest-main' is owned by:
'S-1-5-32-544'
but the current user is:
'S-1-5-21-725345543-602162358-527237240-2219279'
To add an exception for this directory, call:

  git config --global --add safe.directory C:/ProgramData/edkrepo/edk2-edkrepo-manifest-main'  

"""

Workaround:
In the "C:\ProgramData\edkrepo" folder, delete the "edk2-edkrepo-manifest-main" manifest-repo folder, then run "edkrepo manifest" from a non-Administrator Command Prompt to cause the "Cloning global manifest repository" to occur again. After this, both Administrator Command Prompt and non-Administrator Command Prompt can run "edkrepo manifest".

test system configuration:
edkrepo 3.1.0, Python 3.8.10, git version 2.39.1.windows.1, Windows 11

Organize import statements in edkrepo

https://peps.python.org/pep-0008/#imports

We want imports to be strategically separated in the following method:

1. All system imports (packages included in standard Python distribution)
2. All third party imports (packages that needed to be installed via pip)
3. All edkrepo imports

All of these imports should be blocks, with a blank line separating them.

Please submit one PR with one commit for each of the following categories:

• edkrepo/commands/*
• edkrepo/common/*
• All of these together:
  • edkrepo/config/*
  • edkrepo/git_automation/*
  • edkrepo/command_completion_edkrepo.py
  • edkrepo/edkrepo_cli.py
  • edkrepo/edkrepo_entry_point.py
• edkrepo_manifest_parser/edk_manifest.py

Add Windows Container Support to EdkRepo

EdkRepo should be able to install and run inside a Windows container environment. This requires the following changes:

1. Add a /silent parameter that will suppress the display of any GUI elements, message boxes, etc.
2. If no interactive desktop is available, then run in silent/text mode only.

Daisy-chaining PatchSet fails to create branch

Currently, two PatchSet can be chained together to create a dynamic branch.

If a third or more PatchSet is chained in a row, these PatchSet fail to create a dynamic branch when used, and non-gracefully fails when the PatchSet steps are attempted on the wrong branch or commit.

edkrepo 3.1.0

edkrepo checkout with SparseData combination can have wrong sparse files checked out.

If you have two combo ("combo_A" and "combo_B"), both with sparseCheckout="true" on a remote repo ("repo"), but there is a section defined for "combo_A", and a global section used by all combo, the following two cases can happen:

While on "combo_A", running "edkrepo checkout combo_B" will not run the "Resetting sparse checkout state..." step and "Performing sparse checkout..." step.
While on "combo_B", running "edkrepo checkout combo_A" will not run the "Resetting sparse checkout state..." step and "Performing sparse checkout..." step.
-workaround: calling "edkrepo sync" after the 'checkout' command will reset to correct sparse checkout settings.

While on "combo_A", running "edkrepo checkout combo_A" will always cause the "Resetting sparse checkout state..." step and "Performing sparse checkout..." step to be run.
-workaround: none. this just makes the edkrepo checkout command take longer.

test environment: edkrepo 3.1.0, Python 3.8.10, git 2.39.1.windows.1, Windows 11.

Scan for line length violations in edkrepo

https://peps.python.org/pep-0008/#maximum-line-length

There may be areas in the code where some lines are of a very long length (>120 characters). Propose shortening them by breaking into multiple lines via the following protocol:

• Strong preference to use implicit line continuation inside () [] and {}
• Expressions can be wrapped in ()
• Otherwise, default to using a backslash character \ to indicate line break

Please submit one PR with one commit for each of the following categories:

• edkrepo/commands/*
• edkrepo/common/*
• All of these together:
  • edkrepo/config/*
  • edkrepo/git_automation/*
  • edkrepo/command_completion_edkrepo.py
  • edkrepo/edkrepo_cli.py
  • edkrepo/edkrepo_entry_point.py
• edkrepo_manifest_parser/edk_manifest.py

Standardize comments throughout edkrepo

https://peps.python.org/pep-0008/#comments

Need to standardize comments throughout the code:

• Block comments: # + space + comment
  • # example comment
• Inline comments: 2 spaces + # + space + comment
  • print('some code')  # example comment
• Docstrings (comments with # or triple quotes right below a function declaration)
  • One-liner docstrings should be in following format:
    """docstring"""
    • Using triple quotes, be sure to use double quotes " instead of single quotes '
  • Multi-line docstrings should be in following format:
    """
docstring
docstring
"""
    • Also use triple quotes with the double quote char instead of the single quote char, as in previous bullet
  • All instances of ''' should be change to """

Please submit one PR with one commit for each of the following categories:

• edkrepo/commands/*
• edkrepo/common/*
• All of these together:
  • edkrepo/config/*
  • edkrepo/git_automation/*
  • edkrepo/command_completion_edkrepo.py
  • edkrepo/edkrepo_cli.py
  • edkrepo/edkrepo_entry_point.py
• edkrepo_manifest_parser/edk_manifest.py

Update manifest parser to support Python virtual environment configuration on a per combo basis

The manifest parser must be updated to support defining and enabling python virtual environments. To do so the following changes must be implemented:

Update the definition of combos to add the venv_enable and venv_cfg_path data. These should be optional. venv_enable should take True or False

<Combination name="" 
 description= =""
 venv_enable=True|False>
  <Source localRoot="" 
                remote="" 
                branch|tag|sha= =""                     
                sparseCheckout =""               
                venv_cfg=path to req.txt />
</Combination>

Update the combo and repo source properties as follows:

Combination = namedtuple('Combination', ['name', 'description', ‘venv_enable’])

venv_enable returns True|False If not present in manifest default value is false

RepoSource = namedtuple('RepoSource', ['root', 'remote_name', 'remote_url', 'branch', 'commit', 'sparse', 'enable_submodule', 'tag', 'venv_cfg’]

Make usage of blank lines more consistent in edkrepo

https://peps.python.org/pep-0008/#blank-lines

Ensure all code in edkrepo fits the following criteria:

• Surround top-level function and class definitions with two blank lines.
• Method definitions inside a class are surrounded by a single blank line.

In short, all classes should have 2 blank lines surrounding them, all global level functions should have 2 blank lines surrounding them, and all methods inside of a class should have 1 blank line surrounding them.

Please submit one PR with one commit for each of the following categories:

• edkrepo/commands/*
• edkrepo/common/*
• All of these together:
  • edkrepo/config/*
  • edkrepo/git_automation/*
  • edkrepo/command_completion_edkrepo.py
  • edkrepo/edkrepo_cli.py
  • edkrepo/edkrepo_entry_point.py
• edkrepo_manifest_parser/edk_manifest.py

"edkrepo sync" support for tags

edkrepo manifest file Combination and PatchSet can include git tag information, but "edkrepo sync" does not detect or notify user if tags locally and tags on the Remote have somehow deviated from each other.

In some other cases, "edkrepo sync" and "edkrepo sync --fetch" may not create all the same local tags as a fresh 'edkrepo clone' will.

Manually running 'git fetch --tags' on each of the repo seems to work as a workaround for both cases.

edkrepo 3.1.0, git version 2.39.1.windows.1

Improve automatic remote prune in EdkRepo Sync

EdkRepo Sync has heuristics to detect the scenario where changes in the remote branch layout can cause errors when fetching all branches. For example, if one changes a branch name from refs/heads/a/b/c to refs/heads/a/b/c/d, then "c" changes from a file to a directory. This will cause the following error when running fetch:

error: cannot lock ref 'refs/remotes/origin/a/b/c/d': 'refs/remotes/origin/a/b/c' exists; cannot create 'refs/remotes/origin/a/b/c/d'

This change adds an additional heuristic that triggers the automatic
remote prune for the case of "error: cannot lock ref". This should catch more cases of the problem above.

Common Virtual Environment Functions

EdkRepo must provide a set of common methods for configuring and working with Python virtual environments as defined in project manifest files.

This is a pre-requisite for completing #60