jenkins-x / go-scm

Package scm provides a unified interface to multiple source code management systems.

go-scm

A small library with minimal dependencies for working with Webhooks, Commits, Issues, Pull Requests, Comments, Reviews, Teams and more on multiple git provider:

• GitHub
• GitHub Enterprise (you specify a server URL)
• BitBucket Server
• BitBucket Cloud
• GitLab
• Gitea
• Gogs
• Fake
• Azure Devops

Building

See the guide to prerequisites, building and running the code

Working on the code

Clone this repository and use go test...

git clone https://github.com/jenkins-x/go-scm.git
cd go-scm
go test ./...

Writing tests

There are lots of tests for each driver; using sample JSON that comes from the git provider together with the expected canonical JSON.

e.g. I added this test for ListTeams on github: https://github.com/jenkins-x/go-scm/blob/main/scm/driver/github/org_test.go#L83-116

you then add some real json from the git provider: https://github.com/jenkins-x/go-scm/blob/main/scm/driver/github/testdata/teams.json and provide the expected json: https://github.com/jenkins-x/go-scm/blob/main/scm/driver/github/testdata/teams.json.golden

Trying the client on a provider

There are a few little sample programs in scm/factory/examples which are individual binaries you can run from the command line or your IDE.

To test against a git provider of your choice try defining these environment variables:

• GIT_KIND for the kind of git provider (e.g. github, bitbucketserver, gitlab etc)
• GIT_SERVER for the URL of the server to communicate with
• GIT_USER for git user name if using bitbucketclient
• GIT_TOKEN for the git OAuth/private token to talk to the git server

Git API Reference docs

To help hack on the different drivers here's a list of docs which outline the git providers REST APIs

GitHub

• REST API reference: https://developer.github.com/v3/
• WebHooks: https://developer.github.com/v3/activity/events/types/

Bitbucket Server

• REST API reference: https://docs.atlassian.com/bitbucket-server/rest/6.5.1/bitbucket-rest.html
• Webhooks: https://confluence.atlassian.com/bitbucketserver/event-payload-938025882.html

Bitbucket Cloud

• REST API reference: https://developer.atlassian.com/bitbucket/api/2/reference/

Gitlab

• REST API reference: https://docs.gitlab.com/ee/api/

Fake driver for testing

When testing the use of go-scm its really handy to use the fake provider which lets you populate the in memory resources inside the driver or query resources after a test has run.

client, data := fake.NewDefault()

Community

We have a kanban board of stuff to work on if you fancy contributing!

You can also find us on Slack at kubernetes.slack.com:

• #jenkins-x-dev for developers of Jenkins X and related OSS projects
• #jenkins-x-user for users of Jenkins X

go-scm's Issues

Jx3 unable to create repo on gitlab using go-scm

@jstrachan and the team of jx3 have finished migration of jx promote and jx project last week using go-scm for all git repository operations. But when I used the lasted jx-cli to run jx project quickstart --git-user myusername --git-token mytoken --git-kind gitlab --pr-poll-timeout 30m0s, it failed with this error message error: failed to create git repository myusername/projectname: json: cannot unmarshal array into Go value of type gitlab.gl_namespace.

I also have encountered this error: error: failed to validate options: failed to create ScmClient: failed to find git token: could not find git token for git server mygitlaburl when I randomly tried jx project spring --git-kind gitlab.

I would be very grateful if anyone can take a look at this issue.

GitLab should support GraphQL queries

They work differently than GitHub to some extent (no -label:foo equivalent makes me a sad panda), but it exists. See https://docs.gitlab.com/ee/api/graphql/. I'm particularly motivated to do this now for Lighthouse due to https://gitlab.com/gitlab-org/gitlab/-/issues/34527 being fixed so that I can actually implement the Keeper logic properly for GitLab!

Make scm/driver/internal/hmac not internal?

I'd be interested in reusing this code in a project that's already using go-scm.

But, with it being internal, that's "not allowed".

Would you accept a PR that moved this to pkg?

add support for IsCollaborator()

https://github.com/jenkins-x/go-scm/blob/master/scm/driver/stash/repo.go#L107

https://github.com/jenkins-x/go-scm/blob/master/scm/driver/stash/repo.go#L115

it'd be good to add tests for this like the rest of go-scm does.
e.g. I added this test for ListTeams on github:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/org_test.go#L83-116

you then add some real json from the git provider:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/testdata/teams.json
and provide the expected json: https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/testdata/teams.json.golden

Jenkins-X installation fail with Gitlab

We are trying to install Jenkins-X using Gitlab. We are using guidelines mentioned for jx boot
Our Gitlab version is Gitlab enterprise edition 12.3.5-ee

Jx boot fail with the error

error: creating remote for dev environment environment-jx-experiment-dev: duplicating https://github.com/jenkins-x/jenkins-x-boot-config to /environment-jx-experiment-dev: getting repo for https://github.com/jenkins-x/jenkins-x-boot-config: failed to get repository jenkins-x/jenkins-x-boot-config due to: GET https://gitlab-XXXX.TLDR/api/*v3*/repos/jenkins-x/jenkins-x-boot-config: 410 []
error: failed to interpret pipeline file jenkins-x-boot-config/jenkins-x.yml: failed to run '/bin/sh -c jx step verify env' command in directory 'jenkins-x-boot-config', output: ''
$> jx version NAME VERSION jx 2.0.976 Kubernetes cluster v1.14.8 helm client Client: v2.12.2+g7d2b0c7 git 2.18.0 Operating System Mac OS X 10.14.5 build 18F132

As per Gitlab documentation, V3 apis are discontinued : https://docs.gitlab.com/ee/api/v3_to_v4.html

gitlab: changes do not contains previous sha

gitlab only sends a field called oldrev when a new commit is added, instead of filling the changes field like gitlab. This should be fixed to allow two commit trigger on jx3 for example

Repo full name should always be populated

Miss spelling of the `converBranch` func within the gitlab driver webhook pkg.

Issue

func converBranch appears to defy convention by missing the t. Looking at other packages, it looks like the convention and expect namespace should be convertBranch.

Reference

• https://github.com/jenkins-x/go-scm/blob/main/scm/driver/gitlab/webhook.go#L191

add support for ListLabels()

code goes here: https://github.com/jenkins-x/go-scm/blob/master/scm/driver/stash/issue.go#L30

details of adding tests here: https://github.com/jenkins-x/go-scm#writing-tests

Add Gitlab issue webhook support

add support for FindCombinedStatus()

add support for

go-scm/scm/driver/stash/repo.go

Line 99 in 94b6c05

func (s *repositoryService) FindCombinedStatus(ctx context.Context, repo, ref string) (*scm.CombinedStatus, *scm.Response, error) {

it'd be good to add tests for this like the rest of go-scm does.
e.g. I added this test for ListTeams on github:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/org_test.go#L83-116

you then add some real json from the git provider:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/testdata/teams.json
and provide the expected json: https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/testdata/teams.json.golden

Bitbucket Cloud support listing, creating and removing labels on PRs and issues

see

go-scm/scm/driver/bitbucket/pr.go

Line 58 in aa6b3d8

func (s *pullService) ListLabels(context.Context, string, int, scm.ListOptions) ([]*scm.Label, *scm.Response, error) {

Error in Issue Search

I believe there is a code error in the API for issue searches on Github:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/issue.go#L45

This is broken for GitHub Enterprise users because their urls are of the form https://my-gh-instance.com/api/v3. Because the path is given as /search/issues, emphasis on the preceding slash, the resolved url is https://my-gh-instance.com/search/issues.

The fix here is to remove the preceding slash. I haven't checked if there is any other code doing this but those should be fixed as well.

Invalid create date for a GitHub release creating

When I try to create a GitHub release with the following code lines, I found out the field created_at is incorrect.

func (r *GitHub) Release(version, commitish string, draft, prerelease bool) (err error) {
	client := r.getClient()

	releaseInput := &scm.ReleaseInput{
		Title:      version,
		Tag:        version,
		Commitish:  commitish,
		Draft:      draft,
		Prerelease: prerelease,
	}

	// just publish the draft release if it is existing
	release := r.findDraftRelease(version)
	if release != nil {
		releaseInput.Description = release.Description
		releaseInput.Title = release.Title
		_, _, err = client.Releases.Update(context.TODO(), r.repo, release.ID, releaseInput)
	} else {
		_, _, err = client.Releases.Create(context.TODO(), r.repo, releaseInput)
	}
	return
}

See
https://github.com/kubesphere-sigs/ks-releaser/blob/b25af8c05fdc8ddfa0fe811494aa26ae56144e60/controllers/internal_scm/github.go#L24

You can get the following output with the command gh api repos/kubesphere-sigs/ks-releaser/releases

    "node_id": "MDc6UmVsZWFzZTUyNjY2NzA1",
    "tag_name": "v0.0.6",
    "target_commitish": "master",
    "name": "v0.0.6",
    "draft": false,
    "prerelease": false,
    "created_at": "1970-01-01T00:00:00Z",   # this is incorrect
    "published_at": "2021-11-04T08:55:59Z",

The incorrect creation date makes people cannot get the correct latest version. Such as:

See https://github.com/kubesphere-sigs/ks-releaser

Below is the log output, it looks like the system time is correct.

 2021-11-04T11:08:23.965Z	INFO	releaser-resource	validate update	{"name": "ks-releaser-v0.0.6"}

Version

github.com/jenkins-x/go-scm v1.10.10

Gitlab provider don't support pagination in api calls

Hi !
by default, gitlab cuts the API responses to 20 items. This is a problem when we ask to the api for the members of a group / repository that has more than 20 users (the commands thats need check membership of the user don't work for excluded users)

As first approach we can add the per_page parameter with 100 (the gitlab max_per_page default value) to bypass this problem.

In the future we should implement an auto pagination system for this api calls

j

Bitbucket Cloud support for AddCollaborator & IsCollaborator

see the fake implementations here:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/bitbucket/repo.go#L130-L138

Add methof for GitLab repositry delete

go-scm/scm/driver/gitlab/repo.go

Line 409 in e4c033d

return nil, scm.ErrNotSupported

Bitbucket Server List Organisation Support

Bitbucket Server List Organisation Support is not implemented, but is required for activities such as the JX3 CLI for creating projects

Master builds always fail

Because there is no release pipeline in jenkins-x.yml, release builds always fail - it tries to use the go build pack and things go south. =)

Error creating Git Deployment using go-scm

Hi, I have been trying to use the

// Create creates a new deployment.
Create(ctx context.Context, repoFullName string, deployment *DeploymentInput) (*Deployment, *Response, error)

To create the deployment but I am unable to do so, I am getting Validation Failed error and the following response message which I can't figure out

&{9F62:44F5:35C569:3B8506:6033A188 422 map[Access-Control-Allow-Origin:[*] Access-Control-Expose-Headers:[ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset] Content-Length:[241] Content-Security-Policy:[default-src 'none'] Content-Type:[application/json; charset=utf-8] Date:[Mon, 22 Feb 2021 12:20:25 GMT] Referrer-Policy:[origin-when-cross-origin, strict-origin-when-cross-origin] Server:[GitHub.com] Strict-Transport-Security:[max-age=31536000; includeSubdomains; preload] Vary:[Accept-Encoding, Accept, X-Requested-With] X-Accepted-Oauth-Scopes:[] X-Content-Type-Options:[nosniff] X-Frame-Options:[deny] X-Github-Media-Type:[github.v3; format=json] X-Github-Request-Id:[9F62:44F5:35C569:3B8506:6033A188] X-Oauth-Scopes:[admin:enterprise, admin:gpg_key, admin:org, admin:org_hook, admin:public_key, admin:repo_hook, delete:packages, delete_repo, gist, notifications, repo, user, workflow, write:discussion, write:packages] X-Ratelimit-Limit:[5000] X-Ratelimit-Remaining:[4914] X-Ratelimit-Reset:[1613999012] X-Ratelimit-Used:[86] X-Xss-Protection:[1; mode=block]] {0xc00033e420} {0  0 0 0} {5000 4914 1613999012}}

Note: In the same workflow I am able to Delete the deployment so I think other parameters like context and repoFullName are correct. Also I am giving only Ref in the DeploymentInput which according to the the api reference should be the only required filed, I am not sure what I might be doing wrong.
Any help is highly appreciated :)

Bitbucket Server Repo IsCollaborator not working

Attempting to create Lighthouse Triggers with bitbucketserver as SCMProvider.

However, it never identifies anybody as Collaborator.

I assume the error lies here:

go-scm/scm/driver/bitbucket/repo.go

Line 186 in 499f2b2

path := fmt.Sprintf("/2.0/workspaces/%s/permissions/repositories/%s?q=user.account_id=\"%s\"", wsname, reponame, user)

This fits the Bitbucket Cloud API: https://developer.atlassian.com/cloud/bitbucket/rest/api-group-workspaces/#api-user-permissions-workspaces-get

However the Bitbucket Server API seems to work differently:
https://docs.atlassian.com/bitbucket-server/rest/7.18.1/bitbucket-rest.html#idp297

What is the int key in Issues data of the fake driver ?

go-scm/scm/driver/fake/data.go

Lines 5 to 7 in 049a2db

	type Data struct {
	Issues map[int][]*scm.Issue
	OrgMembers map[string][]string

I thought it was the issue ID at first but then the value should be an issue, not an array of issues right ?
Am i missing something ?

GitLab driver does not handle Comment hooks - only partial code support

GitLab comments are sent with an X-Gitlab-Event header of 'Note Hook'. This isn't handled in go-scm/scm/driver/gitlab/webhook.go , though the commentHook struct is present in the code.

IsMember and IsCollaborator for GitHub error when not true

Because status codes greater than 300 result in an error and the code handles all errors the same.

add support for FindRef()

go-scm/scm/driver/stash/git.go

Line 22 in 21b54ba

func (s *gitService) FindRef(ctx context.Context, repo, ref string) (string, *scm.Response, error) {

it'd be good to add tests for this like the rest of go-scm does.
e.g. I added this test for ListTeams on github:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/org_test.go#L83-116

you then add some real json from the git provider:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/testdata/teams.json
and provide the expected json: https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/testdata/teams.json.golden

Do we support "PullRequestService.AddLabel" with Github

I can not find the code implements, could anyone show the code path, thanks

Webhook processing is broken for bitbucket cloud

We are getting error saying failed to parse webhook:Invalid webhook signature in light house. I think webhook payload parsing is broken.

payload attached for refrence
payload.txt

Add labels to PullRequest

FR for adding labels to the pull request type.

I'm happy to take this on to mess around with this library as a precursor to tektoncd/pipeline#1066

Github engine appears to be not using an authenticated call

Hello! I'm trying to use go-scm to create a pull request on a repo. It's very odd, because it appears that the outgoing call is not using a TOKEN.

This is how I'm setting things up:

	// Create the PR
	client, err := factory.NewClientFromEnvironment()
	if err != nil {
		return fmt.Errorf("failed to create scm client: %w", err)
	}

	fmt.Println("Creating pull request with : ", repo, base, branch)
	u := client.BaseURL
	fmt.Println("Url: ", u.String())
	ctx := context.Background()
	request, response, err := client.PullRequests.Create(ctx, repo, &scm.PullRequestInput{
		Title: "PCTL Generated Profile Resource Update",
		Head:  branch,
		Base:  base,
	})

Where repo is Skarlso/pctl-pr-test. And, after adding some debugging here and there in go-scm code, this is the output:

using driver: github and serverURL:
Creating pull request with :  Skarlso/pctl-pr-test main pr16
Url:  https://api.github.com/
2021/04/19 21:13:14 THIS IS THE FULL URL:  https://api.github.com/repos/Skarlso/pctl-pr-test/pulls
2021/04/19 21:13:14 The body:  {"title":"PCTL Generated Profile Resource Update","head":"pr16","base":"main"}

2021/04/19 21:13:14 Request:
2021/04/19 21:13:14 Request:
2021/04/19 21:13:14 Request2:  map[Content-Type:[application/json]]
out:
2021/04/19 21:13:15 failed to create pull request: error while creating pr: Unprocessable Entity

The outgoing request is not authenticated / doesn't have a token header. The very same call with auth header using CURL works:

curl -H "Authorization: token ${GIT_TOKEN}" https://api.github.com/repos/Skarlso/pctl-pr-test/pulls -d '{"title":"PCTL Generated Profile Resource Update","head":"pr13","base":"main"}'

I have GIT_TOKEN defined in the environment where I'm running things. So, what am I doing incorrectly?

Support nested gitlab repositories

We only store the namespace path and not full path, this leads to issues with incorrect org and repo name in lighthouse.

Related to jenkins-x/jx#7981

comment `lgtm` in gitlab but always returns "you cannot LGTM your own PR."

any progress about this issue jenkins-x/lighthouse#1154?

GitLab response codes

Hi, I'm having trouble with a GitLab v13.x instance and the release API call:
https://example.com/api/v4/projects/<owner>%2F<project>/releases/<sometag>

Any tag that does not exist is responded to with a 403 Forbidden, only tags that exist would work.
This ruins the use of IsScmNotFound.

Would a change make sense that considers "Forbidden" errors as not found?

add support for AddLabel()

Any plan to support: azure devops git repos?

Any plan to support: azure devops git repos? https://azure.microsoft.com/en-us/services/devops/repos/

GitHub driver does not handle multiple statuses with the same label (context)

GitHub keeps multiple versions of a check with the same label (context). The API returns the list in reverse chronological order: https://developer.github.com/v3/repos/statuses/#list-statuses-for-a-specific-ref.

The status resource in go-scm does not expose any information that might help distinguish between versions of checks with the same label.

When a client of go-scm receives a list of statuses that contain duplicate labels, it has no way of finding out which one is the most recent one.

I think the best way forward is for the github driver to filter the statuses and only return one (the most recent) for each label.

More context: tektoncd/pipeline#2188

Any plan to support gitee.com

add support for DeleteLabel()

the method implementation goes here: https://github.com/jenkins-x/go-scm/blob/master/scm/driver/stash/issue.go#L39

details of adding tests here: https://github.com/jenkins-x/go-scm#writing-tests

Gitlab MergeRequestHook parser error

Parsing webhooks of object_kind merge_request results in an error.
Failed to parse webhook: json: cannot unmarshal bool into Go struct field .object_attributes.merge_params.force_remove_source_branch of type string

Webhook payload:

{
...
 "object_attributes": {
    ...
    "merge_params": {
      "force_remove_source_branch": true
    },
...
}

So i looked into the implemation and found that the MergeReqeust and PullRequest struct expects a string and not a boolean.

Is it possible that the payload from Gitlab changed with a newer Version? Or is it broken from the begining?

The testdata indicates that Gitlab send "0" and "1" instead of false and true.

Bitbucket cloud does not support Create PullRequest

see

go-scm/scm/driver/bitbucket/pr.go

Line 96 in aa6b3d8

func (s *pullService) Create(ctx context.Context, repo string, input *scm.PullRequestInput) (*scm.PullRequest, *scm.Response, error) {

repository List does not return user private repos

The current implementation of https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/repo.go#L189 or https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/repo.go#L238 does not return private repositories even though we specify a token.

According to github api changes: https://developer.github.com/changes/2019-12-03-internal-visibility-changes/ we can get that by enabling a visibility parameter for an authenticated user using the /user/repos api.

Support comment reactions

GitHub and Gitea seem to have similar APIs. GitLab has an _Award emoji _ API and BitBucket seems to just have a "like" API.
As a first step, it needs to be decided on how to model a go-scm API which is able to abstract over the various different approaches.

support parsing Ping webhooks

we get ping webhooks particularly on github / github enterprise and right now that generates an error - would be good to handle them properly

add support for ListStatus()

go-scm/scm/driver/stash/repo.go

Line 213 in 94b6c05

func (s *repositoryService) ListStatus(ctx context.Context, repo, ref string, opts scm.ListOptions) ([]*scm.Status, *scm.Response, error) {

it'd be good to add tests for this like the rest of go-scm does.
e.g. I added this test for ListTeams on github:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/org_test.go#L83-116

you then add some real json from the git provider:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/testdata/teams.json
and provide the expected json: https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/testdata/teams.json.golden

Bitbucket Cloud create repository in a Team/Organisation

the current code for repository creation defaults to using a workspace named after the current user login:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/bitbucket/repo.go#L79

we probably need some work to be able to support creating a repository in a team (organisation)

Missing the PR review event

In

go-scm/scm/repo.go

Lines 83 to 92 in 893b81a

	HookEvents struct {
	Branch bool
	Issue bool
	IssueComment bool
	PullRequest bool
	PullRequestComment bool
	Push bool
	ReviewComment bool
	Tag bool
	}

Because the review event is missing, jx gitops webhook update configures the PR review events on Github like so:

Which, in turn, means that the review_acts_as_lgtm and review_acts_as_approve options in lighthouse do not work

support parsing github `status` webhooks

Support retrieving commit statuses by state

Currently, go-scm only supports the most recent commit status for a particular context from GitHub, however in some cases it can be useful to see whether a state has ever been posted for a particular context (for example, the following PR could potentially, although unlikely, cause a continuous loop by not knowing whether a success has previously been posted: jenkins-x/lighthouse#1398).

To fix this, I think we should revert back to the original behaviour of this library.

graphql endpoint for Github Enterprise is incorrect

The endpoint should be: http(s)://[hostname]/api/graphql

The currently produced endpoint is: http(s)://[hostname]/api/v3/graphql

See: https://developer.github.com/enterprise/2.18/v4/guides/forming-calls/#the-graphql-endpoint

add support for DeleteRef

go-scm/scm/driver/stash/git.go

Line 26 in 21b54ba

func (s *gitService) DeleteRef(ctx context.Context, repo, ref string) (*scm.Response, error) {

it'd be good to add tests for this like the rest of go-scm does.
e.g. I added this test for ListTeams on github:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/org_test.go#L83-116

you then add some real json from the git provider:
https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/testdata/teams.json
and provide the expected json: https://github.com/jenkins-x/go-scm/blob/master/scm/driver/github/testdata/teams.json.golden

Live/end-to-end test framework and tests for all providers

I'm not comfortable with the fact that our tests for "send a payload to the provider and get a payload in return" are not really testing anything but that we're sending the payload we expect to send and that we can parse the return payload we expect to get back. I'd really like to see live/end-to-end tests, verifying not just that our code will create a payload we expect when we, say, create a pull request, but that the payload is valid for the provider and that we really do get back the response we'd expect. These wouldn't be tests we'd necessarily run all the time, since we'd need accounts, creds, servers for some of the providers, etc, and we'd have to create/manipulate real resources in those providers, but I think there'd be a ton of value in knowing that we're not just guessing that we're generating the right payload and that we know what to do with the response.