ibm / gauge

Measure release insights and recommendations for open-source dependencies. Note: this project is archived.

Note: This project is archived

This project is no longer actively maintained. The codebase will be removed from general availability on January 2.

For other open source security scoring projects, we suggest using scorecard at https://github.com/ossf/scorecard/

Gauge

Measure release insights and get recommendations for open-source dependencies.

About Gauge

For OSS ecosystem, there are established practices for disclosing, discovering and remediating vulnerabilities in the code. Although, in the wake of recent cybersecurity incidents it is becoming important to understand and assess risks associated with developers and their contribution practices. Project “gauge” aims to provides risk assessment for release engineering. For instance, when you upgrade your OSS dependency, it measures risk from every commit that went into the new release, developers that contributed those changes, code review practices observed and types of changes that went into release (performance fix, security fix, but fix, etc.). Core motivation behind project gauge is to bring visibility and auditing into OSS releases.

Requirements

To be able to run this project successfully, the following needs to be configured:

1. First, Golang must be installed. Downloaders can be installed from the Go website located here: https://go.dev/doc/install

2. Set necessary environment variables.

• GITHUB_API_KEY needs to be set as an environment variable with a valid PAT token, with the "public_repo" scope defined. For details on how to set this up, documentation is located here: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token.

We have also setup a cache to avoid making repeated API calls to GitHub. This should help with better performance and get around hitting the API rate limit. Please open an issue to get access to the cache API server. Soon we plan to make it publicly accessible.

• RELEASE_LIB_SERVER (optional): Set the caching API server URL

3. Next, the project can be compiled. To build the project, compile the code with the following command to create the executable gauge:

make

Config file

Copy the .gauge.yaml file and update the control check parameters/thresholds.

## gauge control file

runtime-configs:
  # releaselib service
  releaselib-service: "http://127.0.0.1:9950"

  # weather api key
  # weather apis are used to resolve location information
  # from github to specific country names
  weather-api-key: 

  # github api key
  # github api key is use to avoid rate limit
  github-api-key: 

release-control:
  enable: true
  # maximum release lag for dependencies in terms of
  # their versions 
  max-release-lag: 3

  # maximum release lag for dependencies in terms of
  # time duration (in days)
  max-release-lag-duration: 180

  # Ensure every code change (pull request) has been reviewed
  # by atleast one reviewers (who is different from the author)
  peer-review-enforced: true

  # Zombie changes are the ones that are commited to `main` 
  # branch directly without formal pull request
  # Control to block such code changes
  zombie-commit-enforced: true

Sample Run

Next, its time to try it out!

There are two operational modes for running gauge today.

1. Package: Evaluate health of individual package/repository

2. SBOM : Evaluate health of all OSS packages from the SBOM

Package Mode

Say, you are using python package flask with current version 2.1.1 and you want to evaluate next version before you upgrade to it. You can run following query against gauge to get those insights:

./gauge package -p flask -e python -t 2.1.1 -r https://github.com/pallets/flask

complete log file is available at: /tmp/gauge-075403876
********************************************************************************
Gauge Report for package `flask`
********************************************************************************
Release Measures:
	Current version: 2.1.1
	Latest version: 2.1.2
	Release lag (versions): 1
	Release lag (days): 28 days
--------------------------------------------------------------------------------
		Recommended update
		 Version - 2.1.2
		 Release Time - 2022-04-28 17:48:24 +0000 UTC
		 Num of unique contributors - 7
		 Num of unique reviewers - 0
		 Non peer reviewed changes - 0
		 Num of zombie commits - 15
		 Change annotations - ['docs','typing','testing']
--------------------------------------------------------------------------------                 

In this mode, we have limited feature to discover (if missing) github repsitory path from package-names.

SBOM Mode

This will be shortly available

In this mode, gauge can accept CycloneDX/SPDX formatted SBOM as input and provide evaluation/recommendations for every OSS dependency from SBOM.

WIP

We envision gauge to cover the OSS universe that also includes different modalities of packaging/distributing OSS components:

1. container images
2. Deployment YAMLs (e.g. CRDs, policies, tekton tasks)
3. App bundles (e.g. k8s operators)

We also have an opportunity to enrich our release insights and recommendations with sophistcated ML techniques.

gauge's Issues

No Error Output If Github PAT Invalid

Describe the bug
When using an expired or invalid Github PAT, I was able to run Gauge against SBOMs fully in suspiciously quick times, with no clear error message that the PAT token was invalid or that a 401 Unauthorized was returned when reaching Github API. Instead, during the run many messages of "** contributor stats not available **" appeared, which was not clear to me that it was a Github token issue.

Expected behavior is to get error of 401 Unauthorized for the Github API or invalid token message.

To Reproduce
Steps to reproduce the behavior:

1. set export GITHUB_API_KEY=abcd
2. run Gauge, example: ./gauge package -p flask -e python -t 2.1.1 -r https://github.com/pallets/flask
3. observe run and see that there is no clear error and output is blank and log results are also empty

Expected behavior
Expected behavior is to get error of 401 Unauthorized for the Github API or invalid token message shown.

Desktop (please complete the following information):

• OS: MacOS Monterey 12.4
• go version go1.17.5 darwin/amd64

Releases Feature Not Used for Some Packages

Describe the bug
For packages such as neo4j, their use of Releases has been stale since 2017, though they still update tags with release numbers. This causes Gauge to misidentify the latest release tag name.

Tags link: https://github.com/neo4j/neo4j/tags

To Reproduce
Steps to reproduce the behavior:

1. run gauge with the following command:
   ./gauge package -p neo4j -e unknown -t 0 -r https://github.com/neo4j/neo4j -f output.json

2. View output file with incorrect latest release:

{
"release_report": {
"recommendations": {
"package_name": "neo4j",
"recommended_version": "3.2.0-alpha08",
"release_timestamp": "2017-04-11T08:35:10Z",
"num_uniq_authors": 0,
"num_uniq_reviewers": 0,
"zombie_commits": 0,
"non_peer_reviewed_prs": 0,
"change_annotations": null
},
"insights": {
"package_name": "neo4j",
"current_version": "0",
"repo": "https://github.com/neo4j/",
"latest_version": "3.2.0-alpha08",
"latest_release_timestamp": "2017-04-11T08:35:10Z",
"is_latest": false,
"release_lag": 65,
"major_release_lag": 65,
"release_time_lag": "106751 days",
"annotations": null,
"commit_history": {
"Changes": null,
"Contributors": 0,
"Approvers": 0,
"ZombieChanges": 0
}
}
},

Expected behavior
The latest release should be labeled as "4.4.9" instead of "3.2.0-alpha08".

** Notes **
If a project isn't using the release feature, that may not be a common occurrence and I recommend resolving this issue if this keeps coming up with different projects.

Add support for tags

Is your feature request related to a problem? Please describe.
while running gauge against colors.js repo, discovered that projects might just tag the releases on GitHub source, but not create an explicit release. So, add a support for discovering these tags.

Describe the solution you'd like
Support discoverability of releases through tags. Also, need to make differentiate release tags vs operational tags.

Additional context
When ran against colors.js repo, with additional debug statements:

./gauge package -p colors -e Node -t v1.3.3 -r https://github.com/Marak/colors.js
err GET https://api.github.com/repos/Marak/colors.js/releases/latest: 404 Not Found []

Support JSON Output store

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
Add support in gauge to store results in JSON format

Describe alternatives you've considered

Additional context

Deepscan not available on package type scans

Describe the bug
Deepscan flag is only interpreted on SBOM type scans. When scanning package and defining -d, gauge does not run and instructs to define valid flags

Expected behavior
Gauge should be able to handle running gauge on package scans with "-d true" set as well

Additional context
In function SBOM() in cmd/gauge/cli/sbom.go, the deepscan variable is defined. To fix, define deepscan variable also in cmd/gauge/cli/package.go

Support SBOM input format

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
Currently, gauge accepts package name as input. Add support for it to accept SBOM as input and run gauge against every OSS dependency in the SBOM

Describe alternatives you've considered
None

Additional context
None

Add support to store result in json format

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
Add support in gauge to store results in JSON format

Describe alternatives you've considered

Additional context

Tag Parsing limited to first 30 results

Describe the bug
In pkg/ghapis/release.go, function GetChangeInsights, the repository Tags are looped to get the commit SHA for the current release and the latest release. This function does not fully search all the tags because the query params are set to default, to list 30 (reference to default 30 here.

Expected behavior
To get commit_history accurately, all tags should be searched to compare current vs latest tag and the changes between the two.

Additional context
To fix, loop through the tags and also page through all pages similar to how it is performed in the same file releases.go, GetAllReleases function.