JF_GIT_API_ENDPOINT for GitHub Enterprise about frogbot

[Jetski5822]

and boom - working!

[eyalbe4]

Awesome @Jetski5822!
We may need to add a note to Frogbot's documentation with the above JF_GIT_API_ENDPOINT. We're looking into it right now.
We're working hard on enhancing and improving Forgot. Thanks for using it!

[omerzi]

Hi @Jetski5822, thank you for reporting this issue. Can you please share your YAML configuration? Also, which CI framework are you using in this case?

[Jetski5822]

@omerzi here we go;

I also get this too when JF_GIT_API_ENDPOINT is set to https://github.somecompany.com

[2022-12-14T06:02:41.993Z] Auditing base branch: somerepo release
[2022-12-14T06:02:41.993Z] 06:02:41 [Debug] Created temp working directory: /tmp/jfrog.cli.temp.-1670997761-4037391053
[2022-12-14T06:02:41.993Z] 06:02:41 [Debug] Downloading somecompany/somerepo from branch:<release>
[2022-12-14T06:02:41.993Z] 06:02:41 [Error] EOF
script returned exit code 1

heres the build script;

pipeline {
  agent { label 'docker-xlarge' }
  options {
    skipDefaultCheckout()
    disableConcurrentBuilds(abortPrevious: true)
  }

  environment {
    // [Mandatory only for projects which use npm, pip, pipenv, nuget and dotnet to download their dependencies]
    // The command that installs the project dependencies (e.g "npm i", "nuget restore" or "dotnet restore")
    JF_INSTALL_DEPS_CMD= "dotnet restore"

    // [Mandatory]
    // JFrog platform URL (This functionality requires version 3.29.0 or above of Xray)
    JF_URL= "https://artifactory.somecompany.com"

    // [Mandatory]
    // GitHub accesses token with the following permissions
    //JF_GIT_TOKEN= credentials("GitHub_TOKEN")
    JF_GIT_PROVIDER= "github"

    // [Mandatory]
    // GitHub project namespace
    JF_GIT_OWNER= "somecompany"

    // [Mandatory]
    // GitHub repository name
    JF_GIT_REPO= "somerepo"

    // [Mandatory]
    // API endpoint to GitHub server
    JF_GIT_API_ENDPOINT= "https://github.somecompany.com"

    // Uncomment the below options if you'd like to use them.

    // [Mandatory if JF_USER and JF_PASSWORD are not provided]
    // JFrog access token with 'read' permissions for Xray
    // JF_ACCESS_TOKEN= credentials("JF_ACCESS_TOKEN")

    // [Optional, default: "."]
    // Relative path to the project in the git repository
    // JF_WORKING_DIR="${env.WORKSPACE}"

    // [Optional]
    // Xray Watches. Learn more about them here: https://www.jfrog.com/confluence/display/JFROG/Configuring+Xray+Watches
    // JF_WATCHES= <watch-1>,<watch-2>...<watch-n>

    // [Optional]
    // JFrog project. Learn more about it here: https://www.jfrog.com/confluence/display/JFROG/Projects
    // JF_PROJECT= <project-key>

    // [Optional, default: "FALSE"]
    // Displays all existing vulnerabilities, including the ones that were added by the pull request.
    // JF_INCLUDE_ALL_VULNERABILITIES= "TRUE"

    JENKINS_VERBOSE = 'true'
    BUILD_CONFIGURATION = 'Release'
    JFROG_CLI_LOG_LEVEL = "DEBUG"
    DOTNET_CLI_TELEMETRY_OPTOUT = 'true'
    DOTNET_ROOT = "/usr/share/dotnet/"
  }

   stages {
    stage('Checkout') {
        steps {
          checkout scm
        }
    }

    stage('Download Frogbot') {
      steps {
        // For Linux / MacOS runner:
        sh "curl -fLg https://releases.jfrog.io/artifactory/frogbot/v2/[RELEASE]/getFrogbot.sh | sh"
      }
    }

    stage ('Set Scan Pull Request ENV VARS') {
      when { expression { env.CHANGE_ID != null } }
      steps {
        script {
          env.FROGBOT_CMD="scan-pull-request"
          env.JF_GIT_BASE_BRANCH=env.CHANGE_TARGET
          env.JF_GIT_PULL_REQUEST_ID=env.CHANGE_ID
        }
      }
    }

    stage('dotnet restore') {
      steps {
		sh "dotnet restore \"NicksSolution.sln\" --nologo"
      }
    }

    stage ('Scan Pull Requests') {
      steps {
        withCredentials([string(credentialsId: 'frogbot', variable: 'JF_GIT_TOKEN')]) {
		  withCredentials([usernamePassword(credentialsId: 'jfrogreadonly', usernameVariable: 'JF_USER', passwordVariable: 'JF_PASSWORD')]) {
		    sh "./frogbot ${env.FROGBOT_CMD}"
		  }
        }
      }
    }
  }
}

[omerzi]

Thanks @Jetski5822, from your first comment, it seems that the clone stage of the create-fix-pull-requests failed on the authorization. Can you please verify that the following permissions are granted on the GitHub Personal Access Token:
Read and Write permissions to actions, code scanning alerts, commit statuses, pull requests, security events, and workflows.

Another possibility is that you need to enable the with SSO token authorization:
https://docs.github.com/en/enterprise-cloud@latest/authentication/authenticating-with-saml-single-sign-on/authorizing-a-personal-access-token-for-use-with-saml-single-sign-on

[Jetski5822]

@omerzi ive checked the permissions and added a few more, im now stuck with this error

[2022-12-14T10:16:36.377Z] 10:16:36 [Info] 
[2022-12-14T10:16:36.377Z] Auditing base branch: dashboards-service release
[2022-12-14T10:16:36.377Z] 10:16:36 [Debug] Created temp working directory: /tmp/jfrog.cli.temp.-1671012996-2904215866
[2022-12-14T10:16:36.377Z] 10:16:36 [Debug] Downloading somecompany/somerepo from branch:<release>
[2022-12-14T10:16:36.805Z] 10:16:36 [Error] EOF
script returned exit code 1

thoughts?

[omerzi]

@Jetski5822, it appears the EOF error occurred during the download of the repository from GitHub. To perform these operations on all of our supported VCS, we use froggit-go. The package performs a REST API request to GitHub, downloads the repository as a tar file, and extracts it to a temporary directory.

Either the download failed due to bad parameters in the HTTP request or the untar failed and that is the origin of the error.

I would appreciate it if you could perform the same operation using the cURL command as described in GitHub documentation in order to better understand this issue.

It requires a HOSTNAME (which might be the relevant endpoint) and a token.

When the tar file is opened, check its contents to ensure that they are valid.

It will help us determine if there is a problem with the parameter or if the problem is caused by something else.

In the meantime I believe we should improve our logic in the code and check the response from the request here:

https://github.com/jfrog/froggit-go/blob/master/vcsclient/github.go#LL208C2-L208C2

I'm working on it and will update you whenever the fix is integrated into Frogbot.

Thank you very much for your patience and for helping us improve Frogbot. Let me know if you have any further questions/ideas.

[Jetski5822]

@omerzi cool - will give it a go and get back to you. One question, I have the full repo down already, why not just do a git checkout, instead of downloading the whole repo?

[omerzi]

@Jetski5822, because the failing operation is the archive download as tar and then the tar extraction. So I want to try it on your end, maybe there is something in the configuration that we're missing.

[omerzi]

Hey @Jetski5822, since the GitHub documentation I referred you to requires an additional step, I'll clarify my instructions with three easy cURL commands:

1. Get the GitHub archive link:

curl -v \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  http(s)://HOSTNAME/api/v3/repos/OWNER/REPO/tarball/REF

2. In the response, look for the 302 Found status code, and for the Location header, looks something like that:

< HTTP/2 302 
< server: GitHub.com
< date: Thu, 15 Dec 2022 07:39:55 GMT
< content-type: text/html;charset=utf-8
< content-length: 0
< cache-control: public, must-revalidate, max-age=0
< expires: Thu, 15 Dec 2022 07:39:55 GMT
< location: https://codeload.github.com/owner/repo/legacy.tar.gz/refs/heads/branch
< x-github-api-version-selected: 2022-11-28
< x-ratelimit-limit: 5000
< x-ratelimit-remaining: 5000
< x-ratelimit-reset: 1671093595
< x-ratelimit-used: 0
< x-ratelimit-resource: core
< access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 0
< referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
< content-security-policy: default-src 'none'
< vary: Accept-Encoding, Accept, X-Requested-With

In this case, the relevant URL to download the archive is:
https://codeload.github.com/owner/repo/legacy.tar.gz/refs/heads/branch

3. Download the archive to any preferred location:

curl https://codeload.github.com/owner/repo/legacy.tar.gz/refs/heads/branch > <location>

4. Untar the repository and check its content.

Thanks a lot!

[Jetski5822]

@omerzi here you go...

C:\code\test>curl -v -H "Accept: application/vnd.github+json" -H "Authorization: Bearer <beartoken>" https://github.somecompany.com/api/v3/repos/SomeCompany/somerepo/tarball/REF
*   Trying <ipaddress>:443...
* Connected to github.somecompany.com (<ipaddress>) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
> GET /api/v3/repos/SomeCompany/somerepo/tarball/REF HTTP/1.1
> Host: github.somecompany.com
> User-Agent: curl/7.83.1
> Accept: application/vnd.github+json
> Authorization: Bearer <beartoken>
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: GitHub.com
< Date: Fri, 16 Dec 2022 06:38:39 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 0
< Cache-Control: public, must-revalidate, max-age=0
< expires: Fri, 16 Dec 2022 06:38:39 GMT
< Location: https://github.somecompany.com/_codeload/SomeCompany/somerepo/legacy.tar.gz/REF?token=<token>
< Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< Access-Control-Allow-Origin: *
< X-GitHub-Request-Id: <Guid Request Id>
< Strict-Transport-Security: max-age=31536000; includeSubdomains
< X-Frame-Options: deny
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 0
< Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
< Content-Security-Policy: default-src 'none'
< X-Runtime-Rack: 0.053551
<
* Connection #0 to host github.somecompany.com left intact

C:\code\test>curl  https://github.somecompany.com/_codeload/SomeCompany/somerepo/legacy.tar.gz/REF?token=<token> > C:\code\test\legacy.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    14  100    14    0     0     17      0 --:--:-- --:--:-- --:--:--    17

C:\code\test>

contents of legacy.tar.gz is 404: Not Found

[omerzi]

Thank you @Jetski5822.

In the second GET request, it appears the request fails and you get a 404 error message.
Although we followed the steps GitHub provided in their documentation, there seems to be something missing. Do you think it would be possible for you to contact GitHub support and explain the situation?
It would be greatly appreciated if you could attach the cURL output and the steps you have taken about the cURL command and follow their documentation.
That would be a huge help to us as well. If you can't do that, we will try another way. However, I am very confident they will be able to give us an answer. I want to be informed about what needs to be done to fix this issue as soon as possible.
Thanks a lot in advance!

[Jetski5822]

Hey @omerzi im now away till the 30th - so wont be able to contact GH Support, I can try when im back - unless you have a more direct route.

[eyalbe4]

Hi @Jetski5822,
We could also reach out to GitHub Support, but it's very likely that they'll ask follow-up questions about the environment or ask us to test something. I think it's therefore preferable that the ticket to GitHub Support is opened by you. We'll continue being available here of course and supporting you to make sure this issue is resolved and well understood.
Let us know what you think.

[Jetski5822]

hey @eyalbe4 @omerzi I have pinged the guys that look after the onprem instance for me; however; I ran today with the latest of everything... and it seems to have got further...

[2023-01-04T01:57:47.803Z] 01:57:47 [Info] Auditing SomeRepo release
[2023-01-04T01:57:47.803Z] 01:57:47 [Debug] Created temp working directory:  /tmp/jfrog.cli.temp.-1672797467-2155971289
[2023-01-04T01:57:47.803Z] 01:57:47 [Debug] Downloading SomeOwner/SomeRepo , branch: release to: /tmp/jfrog.cli.temp.-1672797467-2155971289
[2023-01-04T01:57:47.803Z] 01:57:47 [Debug] getting GitHub archive link to download
[2023-01-04T01:57:47.803Z] 01:57:47 [Debug] received archive url: https://github.companyname.com/login?return_to=https%3A%2F%2Fgithub.companyname.com%2Frepos%2FSomeOwner%2FSomeRepo%2Ftarball%2Frelease
[2023-01-04T01:57:47.803Z] 01:57:47 [Error] server response: 406 Not Acceptable
script returned exit code 1

If helps: GitHub Enterprise Server 3.4.7

[Jetski5822]

So I had a read through google/go-github#958, and seems I should change my base URL to be

JF_GIT_API_ENDPOINT= "https://github.somecompany.com/api/v3/"

After doing that, It looks like it might actually go green! - let you know....

[Jetski5822]

So - silly question @eyalbe4 , but, a few questions...

1. Where can I see the report thats generated? is that in JFrog?
2. Can I see more that 3 vulnerabilities in a PR?
3. Should this be run on the base branch first? as the PR is specifying vulnerabilities that probably already exist in main

[omerzi]

Hey @Jetski5822, glad everything worked out for you!

1. The report will be in the pull request itself, for example: #193

2. Yes, there is no limit.

3. Frogbot Scan Pull Request shows only new vulnerabilities that were added in the pull request, not old ones. By setting JF_INCLUDE_ALL_VULNERABILITIES to true, you can see all the vulnerabilities, including older ones that already exist in the branch.

BTW, a new version of Frogbot has been released that uses a config file instead of environment variables (both are supported, but frogbot-config.yml is now recommended as the best practice). Check out the readme file on the Frogbot GitHub page: https://github.com/jfrog/frogbot/blob/master/docs/install-github.md

If you have any further questions, I'm more than willing to hear them. Thank you for your feedback. It is very important to us.

[Jetski5822]

@omerzi awesome, ill move to use the yaml file thanks.

So I see this;

But what's weird is that I have made no changes to packages files or dependencies; the only change is the yaml file to run frogbot. Is this expected?

^^ Proof only 1 file changed in this PR

[omerzi]

Does Frogbot report the pull requests as adding new vulnerabilities even though they are existing ones? Could it be that your Jenkinsfile already has JF_INCLUDE_ALL_VULNERABILITIES set to true?

[Jetski5822]

I don't think so, that environment var is commented out. I'd also expect way more vulnerabilities than 3 too, hence the ask.

I'll move to the new format and see if it makes a difference.

[Jetski5822]

@omerzi so I moved to the new format, and it didnt make a difference. I then switched to show all vulnerabilities as I get this

Which shows a lot of duplicates. I am only showing a screenshot of a small section of the screen, there are way more.

One of the concerns from the engineering is that, the vulnerabilities don't actually tell us where they come from which makes tracking down and fixing them difficult.

[omerzi]

Hey @Jetski5822, thanks for the feedback. The difference between these vulnerabilities is their CVEs, but we recognize that the information may be confusing due to the way it is organized. Our team is working on this and will release something soon that we hope will be more informative and clear. Thank you again, I will keep you updated on this issue once progress has been made.

[Jetski5822]

@omerzi I am Afraid the CVEs are the same :( so looks like a bug to me.

Here is another screenshot with the CVEs (I thought I had put that in the previous screenshot but I must have missed it)

[omerzi]

Thanks for clarifying! we'll take that into account

[eyalbe4]

Thanks so much @Jetski5822!
We're looking into this now.

[Jetski5822]

You guys move fast! Thanks

[omerzi]

@Jetski5822, following our discussion, we released v2.5.6 with a few improvements to the code scanning alerts view and the pull request comments.

There are several improvements, including removing duplicates, organizing each row by its CVE ID, changing the term "Components" to "Direct Dependencies," and so on.

It would be great to hear what you think!

[Jetski5822]

@omerzi Wow - great improvements! - its also way faster - really great job.

A few things.... So - screenshot below is the same build run twice.

1. Took 35 minutes.
2. Took 7 minutes

• The ordering in the Direct Dependencies is different.
• Some from 1 are missing in 2 (no change to branch, or base branch)
• Ordering in Direct Dependencies versions should be against that of Direct Dependencies

[omerzi]

@Jetski5822 Thank you for your feedback!
We will improve the way the direct dependencies appear against their versions.
In regards to the 2nd issue, how did you reproduce it? Is this the same pull request?

[Jetski5822]

@omerzi Same PR, just reran the same PR through Jenkins twice... two different results.

[omerzi]

@Jetski5822, we tried to think of all the possible ways this behavior can occur.
With the includeAllVulnerabilities flag turned off, scan-pull-request does the following:

1. Scan the source pull request branch.
2. Scan the target pull request branch.
3. Comments on the diff vulnerabilities between these branches.

With the 'includeAllVulnerabilities' flag turned on, we only scan the source branch and post the results.
Between two runs on the same PRs, there should be no difference as it only performs an audit as the JFrog CLI does, and it should show the same results as the previous scan if the target or source branch dependencies didn't change.

From your screenshots I can see that System.Private.Uri with version 4.3.0 appears in the first screenshot and not in the second. Maybe the Direct dependency *.Admin.Web.Common changed in the commits between the two scans? Maybe the version has changed?

[Jetski5822]

Hey @omerzi sorry for not getting back sooner;

Nothing had changed between the runs, version etc were exactly the same - literally nothing changed. Ill keep an eye on it and give it a bit more testing this week.