exasol / github-keeper Goto Github PK

View Code? Open in Web Editor NEW

2.0 9.0 1.0 99 KB

Tool for unifying our GitHub repository setup. Superseded by github-issue-adapter

License: MIT License

Go 99.44% Shell 0.56%

exasol-integration

github-keeper's People

Contributors

Stargazers

Watchers

Forkers

rohankumardubey

github-keeper's Issues

Update repo configuration to new team conventions

Update branch protection to make all checks mandatory

Unify-labels fails on new repo

Unifying fails if a label should be renamed to another that already exists.

create-branch-protection fails

The create-branch-protection command fails for small-json-files-test-fixture with

panic: Failed to create branch protection for exasol/small-json-files-test-fixture/main. Cause: PUT https://api.github.com/repos/exasol/small-json-files-test-fixture/branches/main/protection: 422 Validation Failed [{Resource:RequiredStatusCheck Field:context Code:already_exists Message:}]

goroutine 1 [running]:
github.com/exasol/github-keeper/cmd.FixBranchProtectionProblemHandler.createBranchProtection(0xc000080780, 0x7fffc57a9ffc, 0x1d, 0xc000407628, 0x4, 0xc000020380)
        /home/jakob/exasol/github-keeper/cmd/branchProtection.go:56 +0x315
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.checkIfBranchProtectionIsApplied(0x7fffc57a9ffc, 0x1d, 0xc000080780, 0xc0000f0101)
        /home/jakob/exasol/github-keeper/cmd/branchProtection.go:74 +0x42b
github.com/exasol/github-keeper/cmd.glob..func1(0xdb6640, 0xc0000aa5a0, 0x1, 0x2)
        /home/jakob/exasol/github-keeper/cmd/branchProtection.go:23 +0xcf
github.com/spf13/cobra.(*Command).execute(0xdb6640, 0xc0000aa580, 0x2, 0x2, 0xdb6640, 0xc0000aa580)
        /home/jakob/go/pkg/mod/github.com/spf13/[email protected]/command.go:860 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xdb6140, 0x43cf1a, 0xd7b1c0, 0xc000000180)
        /home/jakob/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x35a
github.com/spf13/cobra.(*Command).Execute(...)
        /home/jakob/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/exasol/github-keeper/cmd.Execute()
        /home/jakob/exasol/github-keeper/cmd/root.go:16 +0x31
main.main()
        /home/jakob/exasol/github-keeper/main.go:6 +0x20
exit status 2

github-keeper fails with stacktrace for no obvious reason

Details

github-keeper crashes with no obvious reason.

Steps to reproduce

Setup a projekt-keeper credentials file with outdated/invalid github credentials
Execute github-keeper

Expected behavior

github-keeper is reporting that the credentials are invalid and should be updated.

Actual behavior

github-keeper crashes with a stacktrace

Root Cause

root cause seems to be the usage of expired github credentials (e.g. gh token).

Note

⚠️ Apparently github-keeper is retrieving the credentials from the project-keeper credentials file, this is far from being obvious.

Tools	strategy
release-droid	file `~/.release-droid/credentials`
github-keeper	file `~/.release-droid/credentials`
product-integration-tool-chest / python	file `~/.release-droid/credentials`
product-integration-tool-chest / shell scripts	uses tool `gh`, which evaluates environment variable `GH_TOKEN`
project-overview	environment variable `GITHUB_TOKEN`
project-keeper	(no access to github, yet)
Build scripts, generated by project-keeper	environment variable `GITHUB_TOKEN`

Extract from product-integration-tool-chest

Extract github-keeper from product-integration-tool-chest

Add command for reactivating disabled github actions

Unfeasable branch protection rules

Due to the fact that pushing directly to master/main is prohibited in our repositories, adding branch protection rules which require
the execution of workflows/actions on the following event:

on:
  push:
    branches:
      - main

will render PR's unmergeable (without administrative intervention e.g. deleting the specific branch protection rules).
For example see this PR.

Impact

The project never will be Github-Keeper compliant.

If it is complaint: PR's can't get merged
If it is manually fixed by adjusting the protection rules: The project won't be complaint

Root cause

Github-Keeper does not seem to take triggers of workflow/actions into account correctly.

Make one command configure-repo

Unify the create-labels and create-branch-protection command under a new configure-repo command.

Use 'allow' list for github actions

Github allows us to configure an allow list for actions inside the workflows. By that we can enforce that we use only actions that we reviewed and considered as secure.

Fix-Repo fails on repo with no workflows

Branch protection creation fails for runtime generated matrix builds

Fixing branch protection fails when no protection found

% go run ./... configure-repo script-languages-release-c4 --fix

https://github.com/exasol/script-languages-release-c4
Warning: Failed to parse workflow definition 'https://github.com/exasol/script-languages-release-c4/blob/main/.github/workflows/build.yml'. Probably you use some advanced matrix build features there. Github-keeper will not add the checks from this workflow to the branch protection. Please add them manually. 
Warning: Failed to parse workflow definition 'https://github.com/exasol/script-languages-release-c4/blob/main/.github/workflows/release.yml'. Probably you use some advanced matrix build features there. Github-keeper will not add the checks from this workflow to the branch protection. Please add them manually. 
Warning: Failed to parse workflow definition 'https://github.com/exasol/script-languages-release-c4/blob/main/.github/workflows/test-release.yml'. Probably you use some advanced matrix build features there. Github-keeper will not add the checks from this workflow to the branch protection. Please add them manually. 
panic: Failed to create branch protection for exasol/script-languages-release-c4/main. Cause: PUT https://api.github.com/repos/exasol/script-languages-release-c4/branches/main/protection: 422 Invalid request.

No subschema in "anyOf" matched.
No subschema in "oneOf" matched.
Not all subschemas of "allOf" matched.
For 'anyOf/1', {"strict"=>false} is not a null. []

goroutine 1 [running]:
github.com/exasol/github-keeper/cmd.FixBranchProtectionProblemHandler.createBranchProtection({0x7ff7bfeff6a9?}, {0x7ff7bfeff6a9, 0x1b}, {0xc00020d7d0, 0x4}, 0x7ff7bfeff6a9?)
        /Users/chp/git/github-keeper/cmd/branchProtection.go:40 +0x205
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.CheckIfBranchProtectionIsApplied({{0x7ff7bfeff6a9?, 0xc0000c0008?}, 0xc000144000?}, 0x1e?)
        /Users/chp/git/github-keeper/cmd/branchProtection.go:57 +0x35b
github.com/exasol/github-keeper/cmd.glob..func1(0x18c91e0?, {0xc0000e24a0, 0x1, 0x2?})
        /Users/chp/git/github-keeper/cmd/configureRepo.go:29 +0x15f
github.com/spf13/cobra.(*Command).execute(0x18c91e0, {0xc0000e2460, 0x2, 0x2})
        /Users/chp/go/pkg/mod/github.com/spf13/[email protected]/command.go:860 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0x18c9960)
        /Users/chp/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/chp/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/exasol/github-keeper/cmd.Execute()
        /Users/chp/git/github-keeper/cmd/root.go:16 +0x25
main.main()
        /Users/chp/git/github-keeper/main.go:6 +0x17
exit status 2

Don't always require Sonar for branch protection

Some projects like https://github.com/exasol/tableau-connector/ don't use Sonar.
GK should not require Sonar in the branch protection for these projects.

Remove labels

The following labels have been used in the past but are now replaced by custom fields of the board:

source:exasol
source:external
complexity:medium
complexity:high
complexity:low

Add test for create-branch-protection command

Verify slack web-hook

Verify that all our repos define the slack web-hook

Github keeper fails when .github/workflows contains a directory

Github keeper fails when .github/workflows/ of a repo contains a directory

Migrate github-repo check(s) from product-integration-tool-chest to github-keeper

Implement/Validate that github-keeper supports "all" the aspects of what product-integration-tool-chest supports regarding github features.

Once the feature set is supported by github-keeper the product-integration-tool-chest related code can be removed and the project can reference github-keeper for this functionality.

Related Issues:

exasol/product-integration-tool-chest#40

Enable dependabot and securtiy alerts

Move github-labels script here

Integrate the github-labels script into github-keeper

Automate ticket creation

See for example missing files error_code_config.yml, e.g. exasol/script-languages-release#719

Running GK fails for repo sqlalchemy-exasol

Running GK fails for repo sqlalchemy-exasol:

 ✘ user@host ~/github-keeper $ go run . configure-repo "sqlalchemy-exasol"

https://github.com/exasol/sqlalchemy-exasol
panic: unsupported type float64

goroutine 1 [running]:
github.com/exasol/github-keeper/cmd.WorkflowDefinitionParser.replaceSpecificParameterInJobName({}, {0xc0000c62a0, 0x20}, {0x822360, 0xc000526f98}, 0x4e328d)
        /home/nic/Projects/github-keeper/cmd/workflowDefinitionParser.go:140 +0x16e
github.com/exasol/github-keeper/cmd.WorkflowDefinitionParser.replaceParametersInJobName({}, {0xc0000c62a0, 0x19}, 0x8e8f6a, {0xc000534ab0, 0x3, 0x3}, 0x0)
        /home/nic/Projects/github-keeper/cmd/workflowDefinitionParser.go:116 +0x1e5
github.com/exasol/github-keeper/cmd.WorkflowDefinitionParser.fillJobNameParametersForMatrixBuild({}, {0xc00009c0b8, 0xc000542f10}, {0xc0000c62a0, 0x6f})
        /home/nic/Projects/github-keeper/cmd/workflowDefinitionParser.go:79 +0x325
github.com/exasol/github-keeper/cmd.getJobNames(0xc0005430b0, {})
        /home/nic/Projects/github-keeper/cmd/workflowDefinitionParser.go:60 +0x125
github.com/exasol/github-keeper/cmd.WorkflowDefinitionParser.ParseWorkflowDefinition({}, {0xc000571000, 0xfbe})
        /home/nic/Projects/github-keeper/cmd/workflowDefinitionParser.go:35 +0xca
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.getChecksForWorkflowContent({{0x7ffe4bc4322c, 0x0}, 0xc000188000}, {0xc000571000, 0xfbe}, 0xc000542030)
        /home/nic/Projects/github-keeper/cmd/branchProtection.go:230 +0x167
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.getChecksForWorkflow({{0x7ffe4bc4322c, 0xc000188088}, 0xc000188000}, 0xc000542030)
        /home/nic/Projects/github-keeper/cmd/branchProtection.go:225 +0x85
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.getRequiredChecks({{0x7ffe4bc4322c, 0xc00018a190}, 0xc000188000}, 0x0)
        /home/nic/Projects/github-keeper/cmd/branchProtection.go:204 +0x1ad
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.createProtectionRequest({{0x7ffe4bc4322c, 0x98e4f0}, 0xc000188000}, 0x2a)
        /home/nic/Projects/github-keeper/cmd/branchProtection.go:164 +0xa5
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.CheckIfBranchProtectionIsApplied({{0x7ffe4bc4322c, 0xc000010018}, 0xc000188000}, 0x1e)
        /home/nic/Projects/github-keeper/cmd/branchProtection.go:55 +0x17c
github.com/exasol/github-keeper/cmd.glob..func1(0xca5240, {0xc000063040, 0x1, 0x1})
        /home/nic/Projects/github-keeper/cmd/configureRepo.go:29 +0x15e
github.com/spf13/cobra.(*Command).execute(0xca5240, {0xc000063020, 0x1, 0x1})
        /home/nic/go/pkg/mod/github.com/spf13/[email protected]/command.go:860 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xca59c0)
        /home/nic/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        /home/nic/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/exasol/github-keeper/cmd.Execute()
        /home/nic/Projects/github-keeper/cmd/root.go:16 +0x25
main.main()
        /home/user/github-keeper/main.go:6 +0x17
exit status 2

Add validation for create-branch-protection

Change create-branch-protection so that it validates first and can fix the branch protection additionally.
Also change the fix part that it preserves additional required checks.

Add sonar cloud as required check to branch protection

For java repositories: Add sonar cloud as required check to branch protection

Make github-keeper more end user friendly

An end user whose focus is to fix the issues reported by github-keeper is overloaded with "irrelevant" information. On the other hand essential context information to find and address the issue at hand is missing.

I will illustrate this based on a recent usage error:

Scenario

Check consistency of sqlalchemy-exasol repository.

Run github-keeper

 ✘ user@host ~/github-keeper $ go run . configure-repo "sqlalchemy-exasol"

https://github.com/exasol/sqlalchemy-exasol
panic: unsupported type float64

goroutine 1 [running]:
github.com/exasol/github-keeper/cmd.WorkflowDefinitionParser.replaceSpecificParameterInJobName({}, {0xc0000c62a0, 0x20}, {0x822360, 0xc000526f98}, 0x4e328d)
        /home/nic/Projects/github-keeper/cmd/workflowDefinitionParser.go:140 +0x16e
github.com/exasol/github-keeper/cmd.WorkflowDefinitionParser.replaceParametersInJobName({}, {0xc0000c62a0, 0x19}, 0x8e8f6a, {0xc000534ab0, 0x3, 0x3}, 0x0)
        /home/nic/Projects/github-keeper/cmd/workflowDefinitionParser.go:116 +0x1e5
github.com/exasol/github-keeper/cmd.WorkflowDefinitionParser.fillJobNameParametersForMatrixBuild({}, {0xc00009c0b8, 0xc000542f10}, {0xc0000c62a0, 0x6f})
        /home/nic/Projects/github-keeper/cmd/workflowDefinitionParser.go:79 +0x325
github.com/exasol/github-keeper/cmd.getJobNames(0xc0005430b0, {})
        /home/nic/Projects/github-keeper/cmd/workflowDefinitionParser.go:60 +0x125
github.com/exasol/github-keeper/cmd.WorkflowDefinitionParser.ParseWorkflowDefinition({}, {0xc000571000, 0xfbe})
        /home/nic/Projects/github-keeper/cmd/workflowDefinitionParser.go:35 +0xca
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.getChecksForWorkflowContent({{0x7ffe4bc4322c, 0x0}, 0xc000188000}, {0xc000571000, 0xfbe}, 0xc000542030)
        /home/nic/Projects/github-keeper/cmd/branchProtection.go:230 +0x167
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.getChecksForWorkflow({{0x7ffe4bc4322c, 0xc000188088}, 0xc000188000}, 0xc000542030)
        /home/nic/Projects/github-keeper/cmd/branchProtection.go:225 +0x85
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.getRequiredChecks({{0x7ffe4bc4322c, 0xc00018a190}, 0xc000188000}, 0x0)
        /home/nic/Projects/github-keeper/cmd/branchProtection.go:204 +0x1ad
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.createProtectionRequest({{0x7ffe4bc4322c, 0x98e4f0}, 0xc000188000}, 0x2a)
        /home/nic/Projects/github-keeper/cmd/branchProtection.go:164 +0xa5
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.CheckIfBranchProtectionIsApplied({{0x7ffe4bc4322c, 0xc000010018}, 0xc000188000}, 0x1e)
        /home/nic/Projects/github-keeper/cmd/branchProtection.go:55 +0x17c
github.com/exasol/github-keeper/cmd.glob..func1(0xca5240, {0xc000063040, 0x1, 0x1})
        /home/nic/Projects/github-keeper/cmd/configureRepo.go:29 +0x15e
github.com/spf13/cobra.(*Command).execute(0xca5240, {0xc000063020, 0x1, 0x1})
        /home/nic/go/pkg/mod/github.com/spf13/[email protected]/command.go:860 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xca59c0)
        /home/nic/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        /home/nic/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/exasol/github-keeper/cmd.Execute()
        /home/nic/Projects/github-keeper/cmd/root.go:16 +0x25
main.main()
        /home/user/github-keeper/main.go:6 +0x17
exit status 2

Try to figure out what went wrong

Exit status 2 is not very expressive
In order to figure out whats wrong, the user is forced to scroll up quite a bit
(depending on the stack trace, it may even is not on the current screen)
Error details aren't very helpful from and end user perspective panic: unsupported type float64
-> where was this encountered? file, line, ...

Acceptance criteria

Stack traces are optional (e.g. add --debug flag)
Error messages contain more context, e.g. file name, line number, column where the error
have been encountered. Create a error file e.g. containing stack trace etc. and report to the
user where it can be found ... details see error.txt.
Add --verbose flag which e.g. then gives status information about current and passed checks e.g.:

checking-workflows:
  - x.y.z ✓ 
  - matrix build names ✓
other-checks:
  - branch protection settings ✘ 
     details: failed because of ....

Branch protection rule broken for matrix builds

Github-keeper generates the branch protection rule for github-actions based on the job names.
For matrix build jobs that does not work since job names are different for each matrix run.
Example: https://github.com/exasol/exasol-rest-api/blob/main/.github/workflows/linux-build.yml

Add project keeper to project

Add project keeper to this project as it now supports Go projects

Fix Dependabot warnings

Dependabot reports two issues with this repository:

The branch protection "Restrict who can push to matching branches" doesn't get enabled

Background

Our branch protection guidelines define that "Restrict who can push to matching branches:" is set to "Organization administrators, repository administrators, and users with the Maintain role."
It seems that this is not yet set, see

github-keeper/cmd/branchProtection.go

Line 163 in fe58f0c

Restrictions: nil,

Acceptance Criteria

Branch protection is set as in the guidelines

Branch protection creation fails for repos with float matrix build parameter

configure-repo fails when running on a fresh repo

The configure-repo command fails when run against a fresh repository:

% github-keeper configure-repo metabase-driver

metabase-driver 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13d3084]

goroutine 1 [running]:
github.com/exasol/github-keeper/cmd.BranchProtectionVerifier.CheckIfBranchProtectionIsApplied({{0x7ffeefbffb4d, 0xc000128008}, 0xc0001a8000}, 0x5)
	/Users/chp/go/pkg/mod/github.com/exasol/[email protected]/cmd/branchProtection.go:52 +0xa4
github.com/exasol/github-keeper/cmd.glob..func1(0x189f220, {0xc0001110f0, 0x1, 0x1})
	/Users/chp/go/pkg/mod/github.com/exasol/[email protected]/cmd/configureRepo.go:28 +0x15e
github.com/spf13/cobra.(*Command).execute(0x189f220, {0xc0001110d0, 0x1, 0x1})
	/Users/chp/go/pkg/mod/github.com/spf13/[email protected]/command.go:860 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0x189f720)
	/Users/chp/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/chp/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/exasol/github-keeper/cmd.Execute()
	/Users/chp/go/pkg/mod/github.com/exasol/[email protected]/cmd/root.go:16 +0x25
main.main()
	/Users/chp/go/pkg/mod/github.com/exasol/[email protected]/main.go:6 +0x17

Branch-protection updated added wrong status checks

In https://github.com/exasol/sagemaker-extension[sagemaker-extension](https://github.com/exasol/sagemaker-extension) our PR expect builds which never get executed.

We checked the branch protection was updated by Github-Keeper

Here is a link to a PR in the project:

exasol/sagemaker-extension#43

Validation of workflow files should ignore non-pr relevant files

github-keeper fails validation of workflow files even if these are not relevant for pull requests/branch protection:

Validation Error for .github/workflows/release_droid_upload_github_release_assets.yml: matrix github-action jobs with object parameters and no job name are not supported. Please add a name field to the job that combines the matrix parameters into a more readable name. For example "Build with Go ${{matrix.go}} and Exasol ${{ matrix.db }}"

github-keeper must ignore this kind of problem in non-relevant files.