Hi! just having a look at the code, found this minor typo. It really doesn't seems to affect functionality but just for let you know

Thanks for the tool, very useful!

Feedback from an Enterprise Customer

Branch protection: If there are branch protection rules (in particular branches targeting * patterns) the PR creation and update part fails. Our solution in that regard was to proactively create another rule specific for the generated branch that will allow the update. there is perhaps a better approach, but I couldn't find anything that suggested that this issue was being considered.
Also, the branch to be used for the PR suggestion might need to be created under a path to guarantee the new rule takes place (ghas/ghas-something)

HTTP 403 errors misinterpretation: When reviewing if a repo has had scans in the past, if the process fails due to a permissions error of the token or app, the function was using that fail to skip the repo instead of marking a permissions error, also no logging there as to the specifics of the error (https://github.com/NickLiffen/ghas-enablement/blob/main/src/utils/checkCodeQLEnablement.ts#L28)

Issues being disabled: Given that issue tracking management is not a feature we have available at the org level (you have to go through each repo and switch if you want the tracking or not) then adding a check if issues are enabled in order to be used to communicate with repo owners could avoid surprises. In a nutshell this request https://github.com/NickLiffen/ghas-enablement/blob/main/src/utils/enableIssueCreation.ts#L14 was not enough for that scenario

As a separate idea: perhaps a config where the execution can control if they want both issues and PRs or just one of them could also reduce the confusion.

API limits: the strategies worked, and the token expiration problem also went away, wanted to thank you for that. Wanted to mention that even using the App tokens we hit both rate limits, so clearly a run for more than 1k repos is going to deplete the App API limit which is something that other organizations might face.

Flags in commands: I think that by default the commands being used should strive to reduce the number of time/resources in the clone operation, so reducing the depth and potentially disabling the LFS pull could help other engagements in complete their "enablement runs" faster.

I set up a vagrant instance to test using this project. If run 'yarn run getRepos' it gets an error like
`
$ npm run build && node ./lib/getRepos.js

[email protected] build
npx tsc

src/utils/enableProductOnOrg.ts:25:7 - error TS2345: Argument of type '{ org: string; security_product: "dependency_graph" | "dependabot_alerts" | "dependabot_security_updates" | "advanced_security" | "code_scanning_default_setup" | "secret_scanning" | "secret_scanning_push_protection"; enablement: "enable_all" | "disable_all"; }' is not assignable to parameter of type '{ org: string; security_product: "dependency_graph" | "dependabot_alerts" | "dependabot_security_updates" | "advanced_security" | "secret_scanning" | "secret_scanning_push_protection"; enablement: "enable_all" | "disable_all"; } & RequestParameters'.
Type '{ org: string; security_product: "dependency_graph" | "dependabot_alerts" | "dependabot_security_updates" | "advanced_security" | "code_scanning_default_setup" | "secret_scanning" | "secret_scanning_push_protection"; enablement: "enable_all" | "disable_all"; }' is not assignable to type '{ org: string; security_product: "dependency_graph" | "dependabot_alerts" | "dependabot_security_updates" | "advanced_security" | "secret_scanning" | "secret_scanning_push_protection"; enablement: "enable_all" | "disable_all"; }'.
Types of property 'security_product' are incompatible.
Type '"dependency_graph" | "dependabot_alerts" | "dependabot_security_updates" | "advanced_security" | "code_scanning_default_setup" | "secret_scanning" | "secret_scanning_push_protection"' is not assignable to type '"dependency_graph" | "dependabot_alerts" | "dependabot_security_updates" | "advanced_security" | "secret_scanning" | "secret_scanning_push_protection"'.
Type '"code_scanning_default_setup"' is not assignable to type '"dependency_graph" | "dependabot_alerts" | "dependabot_security_updates" | "advanced_security" | "secret_scanning" | "secret_scanning_push_protection"'.

25 requestParams
~~~~~~~~~~~~~

Found 1 error in src/utils/enableProductOnOrg.ts:25

error Command failed with exit code 2.
`

I assume that this is due to the recent addition of the 'code_scanning_default_setup' option.

When using the script, I wasn't able to apply the CodeQL setting to branches that are not the default branch for their repo.

For example, if I specify develop in the .yaml file it will be applied to repos with a default branch of develop but will also apply the CodeQL setting to the default branch of the other repos listed in the repos.json file.

In addition, it will not apply to the develop branch of that repo if the default branch is not develop.

Sample from codeql-analysis-javascript.yaml file:

on:
  push:
    branches: [develop]
  pull_request:
    # The branches below must be a subset of the branches above
    branches: [develop]
  schedule:
    - cron: "34 14 * * 0"

With the config above in my codeql-analysis-javascript.yaml and with two repos listed in the repos.json file the result is: Both repos will get CodeQL enabled. But only against their default branches, regardless of the default branch name.

I am trying to build this first time and I am getting the below error.
src/utils/worker.ts:27:39 - error TS2339: Property 'length' does not exist on type '{}'.

27 for (orgIndex = 0; orgIndex < repos.length; orgIndex++) {

Do I need to prefil the repos file? I thought the app does that and then I can run the npm run start?

I am running on MacOS, not sure if that is an issue. I had to do npm install --force to npm install it.

Environment

• OS: macOS Ventura 13.0
• Node version: v18.12.1
• git version: 2.37.1 (Apple Git-137.1)

Issue description

When running yarn run start there is an error:

Error: Command failed: git add .github/workflows/codeql-analysis.yml
The following paths and/or pathspecs matched paths that exist
outside of your sparse-checkout definition, so will not be
updated in the index:
.github/workflows/codeql-analysis.yml
hint: If you intend to update such entries, try one of the following:
hint: * Use the --sparse option.
hint: * Disable or modify the sparsity rules.
hint: Disable this message with "git config advice.updateSparsePath false"

The file is indeed present within Desktop/tempGitLocations/python-repo/.github/workflows/codeql-analysis.yml

I see the same error when I try to run the git add command myself, however adding the --sparse flag allows me to successfully add the file to the git stage.

Suggested solution

Add a (potentially optional, depending on environment) --sparse flag as an argument for the git add .github/workflows/codeql-analysis.yml command

Detailed log

gregmohler@gregs-mbp ghas-enablement % yarn run start
yarn run v1.22.19
$ npm run build && node ./lib/enable.js

[email protected] build
npx tsc

ghas:inform Platform detected: darwin +0ms
ghas:inform Currently looping over: 1/2. The org name is: public-stuff +3ms
ghas:inform Currently looping over: 1/1. The repo name is: public-stuff/python-repo +0ms
(node:25512) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
(Use node --trace-warnings ... to show where the warning was created)
ghas:inform Enabled GHAS for python-repo. Status: 200 +549ms
ghas:inform Enabled Secret Scanning for python-repo. Status: 200 +667ms
ghas:inform Has public-stuff/python-repo had a CodeQL scan uploaded? false +639ms
ghas:inform As public-stuff/python-repo hasn't had a CodeQL Scan, going to run CodeQL enablement +1ms
ghas:inform Found default branch on the following repository: python-repo. The default branch is: main +218ms
ghas:inform Found default branch SHA on the following repository: python-repo. The default branch is: e2d207ffe1d963713c21a7f1bd9c9b015706fbe6 +207ms
ghas:inform Branch (ref) created on the following repository python-repo?. The branch reference is: refs/heads/ghas-YZeuZ +237ms
ghas:inform [
ghas:inform {
ghas:inform command: 'rm',
ghas:inform args: [ '-rf', './tempGitLocations' ],
ghas:inform cwd: '/Users/gregmohler/Desktop/'
ghas:inform },
ghas:inform {
ghas:inform command: 'mkdir',
ghas:inform args: [ 'tempGitLocations' ],
ghas:inform cwd: '/Users/gregmohler/Desktop'
ghas:inform },
ghas:inform {
ghas:inform command: 'git',
ghas:inform args: [
ghas:inform 'clone',
ghas:inform '--depth',
ghas:inform '1',
ghas:inform '--filter=blob:none',
ghas:inform '--sparse',
ghas:inform 'https://x-access-token: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@callmegreg-0970b8c44b0307a15.ghe-test.com/public-stuff/python-repo.git'
ghas:inform ],
ghas:inform cwd: '/Users/gregmohler/Desktop/tempGitLocations'
ghas:inform },
ghas:inform {
ghas:inform command: 'git',
ghas:inform args: [ 'checkout', '-b', 'ghas-YZeuZ' ],
ghas:inform cwd: '/Users/gregmohler/Desktop/tempGitLocations/python-repo'
ghas:inform },
ghas:inform {
ghas:inform command: 'mkdir',
ghas:inform args: [ '-p', [Array] ],
ghas:inform cwd: '/Users/gregmohler/Desktop/tempGitLocations/python-repo'
ghas:inform },
ghas:inform {
ghas:inform command: 'cp',
ghas:inform args: [
ghas:inform './bin/workflows/codeql-analysis-python.yml',
ghas:inform '/Users/gregmohler/Desktop/tempGitLocations/python-repo/.github/workflows/codeql-analysis.yml'
ghas:inform ],
ghas:inform cwd: '/Users/gregmohler/Code/ghas-enablement'
ghas:inform },
ghas:inform {
ghas:inform command: 'git',
ghas:inform args: [ 'add', '.github/workflows/codeql-analysis.yml' ],
ghas:inform cwd: '/Users/gregmohler/Desktop/tempGitLocations/python-repo'
ghas:inform },
ghas:inform {
ghas:inform command: 'git',
ghas:inform args: [ 'commit', '-m', '"Commit CodeQL File"' ],
ghas:inform cwd: '/Users/gregmohler/Desktop/tempGitLocations/python-repo'
ghas:inform },
ghas:inform {
ghas:inform command: 'git',
ghas:inform args: [ 'push', '--set-upstream', 'origin', 'ghas-YZeuZ' ],
ghas:inform cwd: '/Users/gregmohler/Desktop/tempGitLocations/python-repo'
ghas:inform },
ghas:inform {
ghas:inform command: 'rm',
ghas:inform args: [ '-rf', './tempGitLocations/' ],
ghas:inform cwd: '/Users/gregmohler/Desktop/'
ghas:inform }
ghas:inform ] +1ms
ghas:inform Executing: rm -rf,./tempGitLocations in /Users/gregmohler/Desktop/ +6ms
ghas:inform +101ms
ghas:inform Executing: mkdir tempGitLocations in /Users/gregmohler/Desktop +1s
ghas:inform +19ms
ghas:inform Executing: git clone,--depth,1,--filter=blob:none,--sparse,https://x-access-token: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@callmegreg-0970b8c44b0307a15.ghe-test.com/public-stuff/python-repo.git in /Users/gregmohler/Desktop/tempGitLocations +1s
ghas:error Cloning into 'python-repo'...
ghas:error +0ms
ghas:inform +2s
ghas:inform Executing: git checkout,-b,ghas-YZeuZ in /Users/gregmohler/Desktop/tempGitLocations/python-repo +1s
ghas:error Switched to a new branch 'ghas-YZeuZ'
ghas:error +1s
ghas:inform +32ms
ghas:inform Executing: mkdir -p,.github/workflows in /Users/gregmohler/Desktop/tempGitLocations/python-repo +1s
ghas:inform +20ms
ghas:inform Executing: cp ./bin/workflows/codeql-analysis-python.yml,/Users/gregmohler/Desktop/tempGitLocations/python-repo/.github/workflows/codeql-analysis.yml in /Users/gregmohler/Code/ghas-enablement +1s
ghas:inform +17ms
ghas:inform Executing: git add,.github/workflows/codeql-analysis.yml in /Users/gregmohler/Desktop/tempGitLocations/python-repo +1s
ghas:inform Whitelist returns: false +31ms
ghas:error Error: Command failed: git add .github/workflows/codeql-analysis.yml
ghas:error The following paths and/or pathspecs matched paths that exist
ghas:error outside of your sparse-checkout definition, so will not be
ghas:error updated in the index:
ghas:error .github/workflows/codeql-analysis.yml
ghas:error hint: If you intend to update such entries, try one of the following:
ghas:error hint: * Use the --sparse option.
ghas:error hint: * Disable or modify the sparsity rules.
ghas:error hint: Disable this message with "git config advice.updateSparsePath false"
ghas:error
ghas:error at ChildProcess.exithandler (node:child_process:412:12)
ghas:error at ChildProcess.emit (node:events:513:28)
ghas:error at maybeClose (node:internal/child_process:1091:16)
ghas:error at Socket. (node:internal/child_process:449:11)
ghas:error at Socket.emit (node:events:513:28)
ghas:error at Pipe. (node:net:313:12) +3s
node:internal/process/promises:288
triggerUncaughtException(err, true /* fromPromise */);
^

Error: Command failed: git add .github/workflows/codeql-analysis.yml
The following paths and/or pathspecs matched paths that exist
outside of your sparse-checkout definition, so will not be
updated in the index:
.github/workflows/codeql-analysis.yml
hint: If you intend to update such entries, try one of the following:
hint: * Use the --sparse option.
hint: * Disable or modify the sparsity rules.
hint: Disable this message with "git config advice.updateSparsePath false"

at ChildProcess.exithandler (node:child_process:412:12)
at ChildProcess.emit (node:events:513:28)
at maybeClose (node:internal/child_process:1091:16)
at Socket.<anonymous> (node:internal/child_process:449:11)
at Socket.emit (node:events:513:28)
at Pipe.<anonymous> (node:net:313:12) {

code: 1,
killed: false,
signal: null,
cmd: 'git add .github/workflows/codeql-analysis.yml',
stdout: '',
stderr: 'The following paths and/or pathspecs matched paths that exist\n' +
'outside of your sparse-checkout definition, so will not be\n' +
'updated in the index:\n' +
'.github/workflows/codeql-analysis.yml\n' +
'hint: If you intend to update such entries, try one of the following:\n' +
'hint: * Use the --sparse option.\n' +
'hint: * Disable or modify the sparsity rules.\n' +
'hint: Disable this message with "git config advice.updateSparsePath false"\n'
}

Node.js v18.12.1
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
gregmohler@gregs-mbp ghas-enablement %

cc @NickLiffen

Hi @NickLiffen ,

I am facing some issues while using this application,

1. First time execution, repos are not listing based on the language filters ( "Found 0 repositories that met the valid criteria in the organisation"), but listing all the repos in our org, again the repos.json file is also empty.
2. Further executions, with or without changing the language filter, few errors are are showing, please review the screenshot.

Please let me know, If am missing something in the steps or do I need to do any clean-up activities prior to each execution?

Regards,
Mebin Thomas

Problem:

Yarn states that this module is incompatible with my version of node. The documentation states

Node v16 or higher installed

I'm running 17 and it fails. I can see that in the package.json it explicitly sets:

"engines": {
    "node": "16"
}

System Info:

System OS: Windows
NodeJS: 17.5.0

Logs:

yarn run getOrgs
yarn run v1.22.15
error [email protected]: The engine "node" is incompatible with this module. Expected version "16". Got "17.5.0"
error Commands cannot run with an incompatible environment.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

yarn -v
1.22.15

node -v
v17.5.0

Expected behavior:

The module should accept engines versioned 16 or higher.

Suggested fix:

"engines": {
    "node": ">=16"
}

Please extend the support for projects running with Python, Java etc..

The current implementation is limited with regards to the use of a GitHub App to serve as the identity, and the default timeout (60 minutes) of the JWT token returned by ocktokit for the GitHub App to authenticate with.

For longer running batches that will last more than 1 hour, the token will expire and start failing the subsequent API calls being made.

Right now, we just include a sample codeql-analysis.yml for JavaScript. It would be great if we could create a directory of templates and then have them called something like codeql-analysis-${language}.yml. We would then match the ${language} found within the process.env.LANGUAGE to the language found within the codeql-analysis-${language}.yml. That would make it more of a scalable solution.

I hit a few issues when trying to run this repo.

1. repo.json was super confusing that I had to copy your sample repos.json before I could load repos with yarn run getRepos
2. Public repositories were filtered out with no option to enable it on
3. Advanced security was force enabled when secretscanning or pushprotection was enabled but public repositories always have AS enabled this caused the script to error out.

Error:

    data: {
      message: 'Advanced security is always available for public repos',
      documentation_url: 'https://docs.github.com/rest/reference/repos#update-a-repository'
    }

My diff to make this work:

diff --git a/src/utils/paginateQuery.ts b/src/utils/paginateQuery.ts
index 804cfc7..6704887 100644
--- a/src/utils/paginateQuery.ts
+++ b/src/utils/paginateQuery.ts
@@ -64,18 +64,13 @@ const getRepositoryInOrganizationPaginate = async (
       const languageCheck = process.env.LANGUAGE_TO_CHECK
         ? name.toLocaleLowerCase() === `${process.env.LANGUAGE_TO_CHECK}`
         : true;
-      const publicRepoCheck =
-        process.env.GHES === "true"
-          ? true
-          : visibility === "PRIVATE" || visibility === "INTERNAL"
-          ? true
-          : false;
+      const publicRepoCheck = visibility === "PRIVATE" || visibility === "INTERNAL"
+          ? false
+          : true;
       return (viewerPermission === "ADMIN" || viewerPermission === null) &&
         isArchived === false &&
         languageCheck &&
-        publicRepoCheck
-        ? true
-        : false;
+        publicRepoCheck;
     });

     inform(
diff --git a/src/utils/worker.ts b/src/utils/worker.ts
index 9a047ec..734a33a 100644
--- a/src/utils/worker.ts
+++ b/src/utils/worker.ts
@@ -8,7 +8,7 @@ import { createPullRequest } from "./createPullRequest.js";
 import { writeToFile } from "./writeToFile.js";
 import { restClient as octokit } from "./clients";
 import { commitFileMac } from "./commitFile.js";
-import { enableGHAS } from "./enableGHAS.js";
+//import { enableGHAS } from "./enableGHAS.js";
 import { enableDependabotAlerts } from "./enableDependabotAlerts";
 import { enableDependabotFixes } from "./enableDependabotUpdates";
 import { enableIssueCreation } from "./enableIssueCreation";
@@ -49,9 +49,9 @@ export const worker = async (): Promise<unknown> => {
       const [owner, repo] = repoName.split("/");

       // If Code Scanning or Secret Scanning need to be enabled, let's go ahead and enable GHAS first
-      enableCodeScanning || enableSecretScanning
+      /*enableCodeScanning || enableSecretScanning
         ? await enableGHAS(owner, repo, client)
-        : null;
+        : null;*/

       // If they want to enable Dependabot, and they are NOT on GHES (as that currently isn't GA yet), enable Dependabot
       enableDependabot && process.env.GHES != "true"

Anyway thanks for the repo saved me time and it's all working with my hacks

I'd like to query a subset of the repos in our GH Enterprise set of repositories by using a prefix, for example bh-repos-...

@NickLiffen, can you remind me where in my .env file I paste that prefix to limit my GH repo query?

Hi @NickLiffen 👋

Some feature requests to help us onboard on a per-team basis:

1. In step one: make it possible to gather repos by team rather than language (https://docs.github.com/en/rest/reference/teams#list-team-repositories)
2. Set the branch name in codeql-analysis.yml based on the repo's default branch name

We want to use this app with GitHub enterprise and have registered+installed this as a GitHub App under one of our organisations.

The first thing we notice when we try to use GitHub App authentication is that the .env.sample has a key "APP_ID" while the code refers to the key "GITHUB_APP_ID". However, if we change that and we run the application we run into another issue.

The example below gives an idea of which values we entered in our .env file (secrets have been replaced with similar strings of the same length). We retrieved the installation id from the installation URL (https://github.com/<our_organisation>/settings/installations/12345678).

GITHUB_APP_ID=123456
APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\nMIIEowI ... \n-----END RSA PRIVATE KEY-----"
APP_INSTALLATION_ID=12345678
APP_CLIENT_ID=Iv1.7h9ldim2lpmxdryu
APP_CLIENT_SECRET=lgvtg1hzekulfcabat3gn1zksy38xkvf950px

When we try to list the repositories with the above configuration we run into the following issue:

jorsmat@NB-01514:~/repos/ghas-enablement$ node ./lib/src/getRepos.js
  ghas:inform Could not find file: ./bin/organizations.json. Assuming no organizations have been collected. +0ms
  ghas:inform Collecting repositories for liantisit-infra +2ms
  ghas:inform This is org number 1 of 1 +0ms
Error within function (githubAuth) [@octokit/auth-app] installationId option is required for installation authentication.
Error within function (graphQLClient) We failed to generate a token from the credentials provided on the GitHub App. Please re-check the credentails provided.
  ghas:error Error: We failed to generate the graphql Client
  ghas:error     at graphQLClient (/home/jorsmat/repos/ghas-enablement/lib/src/utils/clients/graphql.js:20:15)

If we run the script using a PAT it works as expected.

As an org admin (or team admin) running this script, the language script can be quite broad and return repositories where the language is only used across 1% of the repository, this is inefficient and not relevant. I would like only to return the languages that are primary, so I know the pull requests I create can be targeted and precise.

Instead of authing from a GitHub PAT, it would be better to auth from a GitHub App.

As a developer on a project where the pull request has been proposed, I would like to be informed a little about what to do with the file, so I can take appropriate action(s). To some developers, the pull request may be confusing and without any detailed information, it may be hard to comprehend and actually do something with. It would be nice if there was a correlating issue that could go alongside the pull request that enables people to know more.

The README.md says in PART 2 that "A pull request gets created on that repository with the codeql-analysis.yml found in the root of this repository." I am not seeing that codeql-analysis.yml file in the root of the this repo.

Hello,

I work at an enterprise that divided it's repositories over multiple organizations in GitHub. We prefer to use this tool with the GitHub App Auth authentication over PAT (private access token) .

Currently with a PAT that has been authorized on all your organizations, you can already run the getOrgs.js script and set GHAS features over multiple organizations with a single run of getRepos.js and enable.js. We would like similar functionality when using GitHub app authentication.
When you use GitHub App authentication this feature is currently not available, because the tool fetches a token for one specific APP_INSTALLATION_ID/organization.

Expected:
Given that the GITHUB_ENTERPRISE and no GITHUB_ORG has been set in the environment.

1. The tool authenticates using the APP_ID and the APP_PRIVATE_KEY.
2. The tool fetches information about the Organizations where the GitHub App has been installed.
3. The tool loops over those organizations (I believe acquiring some kind of token for each one of them) and lists all relevant repos and applies changes to them.

Not expected:

• Filtering organizations is not a requirement for us, because we can already "suspend" a GitHub app installation in the UI if we want to temporarily not use this tool on a certain organization.

Kind regards,
Jors

yarn run build fails on initial clone

Creating repos.json is not optional - required for build

I've got a modification to improve the throughput for my use case, which is just for committing a new workflow to repos. I was wondering if this change would be appreciated by others:

in src/utils/commands.ts: clone with options --depth 1, --filter=blob:none and --sparse. By cloning this way, the amount of data will be reduced. Not eliminated, but reduced.

    {
      command: "git",
      args: ["clone","--depth","1","--filter=blob:none","--sparse", `${baseURL}/${owner}/${repo}.git`],
      cwd: `/Users/${user}/${destDir}/${tempDIR}`,
    },

These pages explain the concepts and I combined a couple. Seemed to give the smallest size and allows adding a new workflow file without issues for me.
https://unix.stackexchange.com/questions/233327/is-it-possible-to-clone-only-part-of-a-git-project
https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/

I can submit a PR if you think it's good. I believe new git >2.19 will support them, but didn't test any git versions except Mac git 2.32 which I am on.

As a consumer of this script, I do not want to be hit with a pull request for a codeql-analysis.yml file if I am already using CodeQL. So, I would like this script to check if there has already been a codeql upload, and if so, do not include my repository within the repos.json.

Hello,

After some testing my colleague noticed an issue with the pull requests for the CodeQL. It looks like a bug.

Expected behaviour

This script only returns repositories where CodeQL results have not already been uploaded to code scanning. If any CodeQL results have been uploaded to a repositories code scanning feature, that repository will not be returned to this list. The motivation behind this is not to raise pull requests on repositories where CodeQL has already been enabled.

Actual behaviour

Initially everything works as expected, ghas-enablement creates a pull request on our repository to add a CodeQL analysis workflow, we merge it, the CodeQL scan runs, but then on the next run of ghas-enablement, a new pull-request seems to get created.

Test environment

• Linux github runner
• Version 3.3.0 of the code

Further details

As far as I can tell from searching on the code, the code that was foreseen to implement the expected behaviour is not yet being triggered (the code in checkCodeQLEnablement.ts).

There are also some duplicate types that can probably be removed, as they do not seem to be used.

export type listCodeScanningParameters =
  Endpoints["GET /repos/{owner}/{repo}/code-scanning/analyses"]["parameters"];

export type listCodeScanningResponse =
  Endpoints["GET /repos/{owner}/{repo}/code-scanning/analyses"]["response"];

I suppose we need to adjust the code so that this code gets called and perhaps also supplement this check (someone might remove the codeql-analysis after initial results were published).

Not sure how to label an issue as being a bug or feature :) perhaps only you can do that @NickLiffen

Hi Nick

We are using the last option with action workflow file when i run the action we get the following issue any help would be highly appreciated

Run npm run getRepos

> [email protected] getRepos
> npm run build && node ./lib/getRepos.js


> [email protected] build
> npx tsc

2022-12-01T12:42:20.944Z ghas:inform Could not find file: ./bin/organizations.json. Assuming no organizations have been collected.
2022-12-01T12:42:20.945Z ghas:inform Collecting repositories for XXXX
2022-12-01T12:42:20.945Z ghas:inform This is org number 1 of 1
2022-12-01T12:42:21.689Z ghas:error HttpError: Not Found
    at /home/runner/work/ghas-enablement/ghas-enablement/node_modules/@octokit/request/dist-node/index.js:86:21
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async getInstallationAuthentication (/home/runner/work/ghas-enablement/ghas-enablement/node_modules/@octokit/auth-app/dist-node/index.js:280:7)
    at async hook (/home/runner/work/ghas-enablement/ghas-enablement/node_modules/@octokit/auth-app/dist-node/index.js:449:7)
    at async Job.doExecute (/home/runner/work/ghas-enablement/ghas-enablement/node_modules/bottleneck/light.js:405:[18](https://github.com/HDRUK/ghas-enablement/actions/runs/3592684492/jobs/6048693035#step:4:19))

Thanks
reuben

• Enable GHAS on each repository before creating the codeql-analysis file.

Summary

Running the GitHub Action step for C# does not work since it does not create PRs to include the CodeQL analysis file for every C# repo. The expectation is when running npm run start it will do its GHAS enablement regardless of the language. Also, I want to mention that this is the only language so far that doesnt do its enablement for my organization.

Code

jobs:
  enable-security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          repository: submittable/ghas-enablement

      - name: Get dependencies and configure
        run: |
          yarn
          git config --global user.name "ghas-enablement"
          git config --global user.email "[email protected]"
      
      - name: Enable security on organization (csharp)
        run: |
          npm run getOrgs
          npm run getRepos
          npm run start
        env:
          LANGUAGE_TO_CHECK: "csharp"
          TEMP_DIR: ${{ github.workspace }}

Logs

2023-03-22T23:13:22.265Z ghas:inform All repos collected. Writing them to file: ./bin/repos.json
2023-03-22T23:13:22.266Z ghas:inform Success created repos.json

> [email protected] start
> npm run build && node ./lib/enable.js


> [email protected] build
> npx tsc
2023-03-22T23:13:27.268Z ghas:inform Platform detected: linux

2023-03-22T23:13:27.273Z ghas:inform Currently looping over: 1/1. The org name is: submittable

I thought there was a way to run the script and toggle Code Scanning to be enabled without requiring a PR approval. Can anyone remind me the setting in the .env or repos.json config files to make the enablement and the commit of the .github/worksflows/codeql.yml happen upon execution of the script?

As an end-user of this utility; I would like to be able to run this on Windows, and not just mac, so I can use the tool without getting errors.

The way that I would go about implementing is:

1. Add a small function that detects the OS of the user. E.G (windows, linux, windows, etc)

2. Add the windows commands here.

3. Use the function that has been created in step 1) to detect what OS the end user is using. If they are using Mac, send to the mac commands, if they are using Windows, send to the Windows commands 👍 If they are using another os like Linux, etc, I would throw an error

^^^ This can be changed, this is just my thoughts, however, 100% welcome other thoughts on this if people have any 👍 Very open for discussions and thoughts 👍

Adding support for Codeowner as an optional flag so CodeQL or other security tools need to have a particular team (security teams typically) approve changes to that file.

Main use case is for developers teams not disabling Code Scanning if its a requirement.

I believe this is a relatively new GitHub feature. It also looks relatively straightforward to add it, since it can be enabled through the endpoint used to already enable secret scanning itself.
https://docs.github.com/en/enterprise-cloud@latest/rest/repos/repos#update-a-repository

I guess there is a related todo in the code that might be picked up as well:

//TODO: I can combine this function and the function found within `enableGHAS` and pass in the `secret_scanning` or `ghas` as a var
export const enableSecretScanningAlerts = async (

Hey, I'm having an issue getting repos.json populated due toghas:error TypeError: Cannot read properties of undefined (reading 'includes'), and was curious if anyone had seen this or is familiar with the issue? Output and .env are below:

# AUTH: GitHub PAT Configuration
GITHUB_API_TOKEN=<gh_tkn>

# Set one of the GITHUB_ENTERPRISE or GITHUB_ORG variables.
GITHUB_ENTERPRISE=
GITHUB_ORG=<OurOrg>

# If you are filtering by language, set the language here. Please use either: javascript, typescript, go, python, ruby, c#, c++, java, or kotlin
LANGUAGE_TO_CHECK=javascript

# Debug Configuration
DEBUG=ghas:*

# GHES Configuration
GHES=false
GHES_SERVER_BASE_URL=

# Temp working directory. This path needs to already exist and follow linux style paths. c:\ghas\tmp == ghas/tmp
TEMP_DIR=<my_path> 

% yarn run getRepos
yarn run v1.22.19
warning ../../package.json: No license field
$ npm run build && node ./lib/getRepos.js

> [email protected] build
> npx tsc

  ghas:inform Could not find file: ./bin/organizations.json. Assuming no organizations have been collected. +0ms
  ghas:inform Collecting repositories for <OurOrg> +1ms
  ghas:inform This is org number 1 of 1 +0ms
  ghas:inform Repo Name: <x> Permission: <x> Archived: false Language: JavaScript Visibility: PRIVATE +1s
.
.
.
  ghas:inform Repo Name: <x> Permission: <x> Archived: true Language: Python Visibility: PRIVATE +0ms
  ghas:inform Found 66 repositories that met the valid criteria in the organisation <OurOrg>. Out of 100. +0ms
  ghas:error TypeError: Cannot read properties of undefined (reading 'includes')
  ghas:error     at getRepositoryInOrganizationPaginate (/Users/mattb/workspace/ghas-enablement/lib/utils/paginateQuery.js:45:20)
  ghas:error     at async paginateQuery (/Users/mattb/workspace/ghas-enablement/lib/utils/paginateQuery.js:73:22)
  ghas:error     at async collectRepos (/Users/mattb/workspace/ghas-enablement/lib/utils/collectRepos.js:21:40)
  ghas:error     at async start (/Users/mattb/workspace/ghas-enablement/lib/getRepos.js:34:9) +0ms
  ghas:error TypeError: Cannot read properties of undefined (reading 'includes')
  ghas:error     at getRepositoryInOrganizationPaginate (/Users/mattb/workspace/ghas-enablement/lib/utils/paginateQuery.js:45:20)
  ghas:error     at async paginateQuery (/Users/mattb/workspace/ghas-enablement/lib/utils/paginateQuery.js:73:22)
  ghas:error     at async collectRepos (/Users/mattb/workspace/ghas-enablement/lib/utils/collectRepos.js:21:40)
  ghas:error     at async start (/Users/mattb/workspace/ghas-enablement/lib/getRepos.js:34:9) +0ms
  ghas:error TypeError: Cannot read properties of undefined (reading 'includes')
  ghas:error     at getRepositoryInOrganizationPaginate (/Users/mattb/workspace/ghas-enablement/lib/utils/paginateQuery.js:45:20)
  ghas:error     at async paginateQuery (/Users/mattb/workspace/ghas-enablement/lib/utils/paginateQuery.js:73:22)
  ghas:error     at async collectRepos (/Users/mattb/workspace/ghas-enablement/lib/utils/collectRepos.js:21:40)
  ghas:error     at async start (/Users/mattb/workspace/ghas-enablement/lib/getRepos.js:34:9) +0ms
  ghas:error TypeError: Cannot read properties of undefined (reading 'includes')
  ghas:error     at getRepositoryInOrganizationPaginate (/Users/mattb/workspace/ghas-enablement/lib/utils/paginateQuery.js:45:20)
  ghas:error     at async paginateQuery (/Users/mattb/workspace/ghas-enablement/lib/utils/paginateQuery.js:73:22)
  ghas:error     at async collectRepos (/Users/mattb/workspace/ghas-enablement/lib/utils/collectRepos.js:21:40)
  ghas:error     at async start (/Users/mattb/workspace/ghas-enablement/lib/getRepos.js:34:9) +0ms
✨  Done in 3.56s.`

Note I'm not done with my test, it's just the end of the day for me right now and need to head out. I'll come back to this with anything more I find. Hopefully you'll do a better job at getting to the root of it than I have. Thanks!

Problem:

When attempting to run yarn run start after getOrgs & getRepos in PowerShell I received the following error:
Error: spawn mkdir ENOENT

When attempting to run yarn run start after getOrgs & getRepos gitBash I received the following error:
Error: spawn git ENOENT
Detailed logs below

System Info:

System OS: Windows
NodeJS: 17.5.0

Condig:

.env Config:
GITHUB_API_TOKEN=ghp_
GITHUB_ENTERPRISE=<private_company_name>
LANGUAGE_TO_CHECK=java
ENABLE_ON=secretscanning,dependabot,dependabotupdates,codescanning
CREATE_ISSUE=true

All other configs left at default

Logs:

When running in PowerShell in Windows 10

node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^

Error: spawn mkdir ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:476:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -4058,
  code: 'ENOENT',
  syscall: 'spawn mkdir',
  path: 'mkdir',
  spawnargs: [ '-p', 'tempGitLocations' ],
  cmd: 'mkdir -p tempGitLocations',
  stdout: '',
  stderr: ''
}

Node.js v17.5.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

When running in git-bash in Windows 10

node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^

Error: spawn git ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:476:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -4058,
  code: 'ENOENT',
  syscall: 'spawn git',
  path: 'git',
  spawnargs: [ 'checkout', '-b', 'ghas-Zigxs' ],
  cmd: 'git checkout -b ghas-Zigxs',
  stdout: '',
  stderr: ''
}

Node.js v17.5.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Expected behavior:

The GHAS tool should differentiate between CMD & PowerShell or specify one be used over the other.

I started to write out the commands here but quickly realized that's a lot to add lol. Suffice to say that though PowerShell can execute some commands from CMD, it seems to do a translation.
Example:
mkdir - Works in either case as there are no argument

rmdir - Doesn't translate over correctly.

• CMD: rmdir /s /q <path> - works fine

• PowerShell: rmdir /s /q <path> => Remove-Item: A positional parameter cannot be found that accepts argument '/q'.

Another I can see is rm. This isn't a command accepted in CMD but it is in PowerShell.

Troubleshooting:

In the end it appears that spawn is having issues detecting the commands to run. I know it's old, but it sees to still be an issue... maybe???
nodejs/node-v0.x-archive#2318

In short:

At the moment child_process.spawn() can only run exe files. This is a limitation of the CreateProcess API.

I tested this with the following:

..\ghas-enablement\serc\utils\commands.ts ->

const commands = [
    {
      command: "cmd",
      args: ["/s", "/c", "mkdir", "-p", `${tempDIR}`],
      cwd: `/Users/${winUser}/${windestDir}`,
    },
...

This actually worked (in PowerShell, but not CMD). I tried several permutations prior to this with no success.

Do you see the same error in your testing?

Problem:

npx fails to install typescript

System Info:

System OS: Windows
NodeJS: 17.5.0

Logs:

yarn run getOrgs
yarn run v1.22.15
$ npm run build && node ./lib/src/getOrgs.js

[email protected] build
npx tsc

Need to install the following packages:
tsc
Ok to proceed? (y) y

This is not the tsc command you are looking for

To get access to the TypeScript compiler, tsc, from the command line either:

• Use npm install typescript to first add TypeScript to your project before using npx
• Use yarn to avoid accidentally running code from un-installed packages
  error Command failed with exit code 1.
  info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Expected behavior:

Yarn will successfully install typescript

Summary

Running the GitHub Action step for Python fails after running it the first time. It ran once and then rerunning it causes it to fail. I plan to put this into a cronjob but manual invocation does not work.

Code

jobs:
  enable-security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          repository: submittable/ghas-enablement

      - name: Get dependencies and configure
        run: |
          yarn
          git config --global user.name "ghas-enablement"
          git config --global user.email "[email protected]"
      
      - name: Enable security on organization (python)
        run: |
          npm run getOrgs
          npm run getRepos
          npm run start
        env:
          LANGUAGE_TO_CHECK: "python"
          TEMP_DIR: ${{ github.workspace }}

Error

2023-03-23T17:01:24.574Z ghas:inform 
2023-03-23T17:01:25.575Z ghas:inform Executing:  git checkout,-b,ghas-xyCIz in //home/runner/work/github-actions/github-actions/tempGitLocations/<REPO>
2023-03-23T17:01:25.581Z ghas:error Switched to a new branch 'ghas-xyCIz'

2023-03-23T17:01:25.581Z ghas:inform 
2023-03-23T17:01:26.582Z ghas:inform Executing:  mkdir -p,.github/workflows in //home/runner/work/github-actions/github-actions/tempGitLocations/<REPO>
2023-03-23T17:01:26.588Z ghas:inform 
2023-03-23T17:01:27.589Z ghas:inform Executing:  cp ./bin/workflows/codeql-analysis-python.yml,//home/runner/work/github-actions/github-actions/tempGitLocations/<REPO>/.github/workflows/codeql-analysis.yml in /home/runner/work/github-actions/github-actions
2023-03-23T17:01:27.598Z ghas:inform 
2023-03-23T17:01:28.599Z ghas:inform Executing:  git add,.github/workflows/codeql-analysis.yml in //home/runner/work/github-actions/github-actions/tempGitLocations/<REPO>
2023-03-23T17:01:28.606Z ghas:inform 
2023-03-23T17:01:29.608Z ghas:inform Executing:  git commit,-m,"Commit CodeQL File" in //home/runner/work/github-actions/github-actions/tempGitLocations/<REPO>
2023-03-23T17:01:29.615Z ghas:inform Whitelist returns: false
2023-03-23T17:01:29.615Z ghas:error Error: Command failed: git commit -m "Commit CodeQL File"

    at ChildProcess.exithandler (node:child_process:419:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1091:16)
    at Socket.<anonymous> (node:internal/child_process:449:11)
    at Socket.emit (node:events:513:28)
    at Pipe.<anonymous> (node:net:322:12)
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

Error: Command failed: git commit -m "Commit CodeQL File"

    at ChildProcess.exithandler (node:child_process:419:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1091:16)
    at Socket.<anonymous> (node:internal/child_process:449:11)
    at Socket.emit (node:events:[513](https://github.com/submittable/github-actions/actions/runs/4495431566/jobs/7926103435#step:4:514):28)
    at Pipe.<anonymous> (node:net:322:12) {
  code: 1,
  killed: false,
  signal: null,
  cmd: 'git commit -m "Commit CodeQL File"',
  stdout: 'On branch ghas-xyCIz\nnothing to commit, working tree clean\n',
  stderr: ''
}

Node.js v18.15.0
Error: Process completed with exit code 1.

As an end-user, I would like better documentation on how to handle GHES with this tool 👍

It would be nice to enable Dependabot version updates as well (perhaps as a switch?) along with Dependabot alerts and Dependabot security updates.

Right now, only Dependabot alerts are enabled if ENABLE_ON=dependabot.

It would also be nice to enable Dependabot security updates as well (perhaps as a switch?)

Problem:

yarn run getOrgs returns error: TypeError: Cannot read properties of null (reading 'organizations')

Please excuse all the sanitation, my company's shy.

System Info:

System OS: Windows
NodeJS: 17.5.0

.env Config:

GITHUB_API_TOKEN=ghp_
GITHUB_ENTERPRISE=<private_company_name>

All other configs are default

Logs:

TypeError: Cannot read properties of null (reading 'organizations')
at performOrganisationsQuery (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:9:103)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async getOrganisationsInEnterprise (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:19:49)
at async index (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:43:22)
at async start (C:\Users\dthornton\code\ghas-enablement\lib\src\getOrgs.js:30:9)
ghas:error TypeError: Cannot read properties of null (reading 'organizations')
ghas:error at performOrganisationsQuery (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:9:103)
ghas:error at processTicksAndRejections (node:internal/process/task_queues:96:5)
ghas:error at async getOrganisationsInEnterprise (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:19:49)
ghas:error at async index (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:43:22)
ghas:error at async start (C:\Users\dthornton\code\ghas-enablement\lib\src\getOrgs.js:30:9) +0ms
TypeError: Cannot read properties of null (reading 'organizations')
at performOrganisationsQuery (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:9:103)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async getOrganisationsInEnterprise (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:19:49)
at async index (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:43:22)
at async start (C:\Users\dthornton\code\ghas-enablement\lib\src\getOrgs.js:30:9)
ghas:error TypeError: Cannot read properties of null (reading 'organizations')
ghas:error at performOrganisationsQuery (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:9:103)
ghas:error at processTicksAndRejections (node:internal/process/task_queues:96:5)
ghas:error at async getOrganisationsInEnterprise (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:19:49)
ghas:error at async index (C:\Users\dthornton\code\ghas-enablement\lib\src\utils\getOrganisationsInEnterprise.js:43:22)
ghas:error at async start (C:\Users\dthornton\code\ghas-enablement\lib\src\getOrgs.js:30:9) +2ms
Done in 5.71s.

Expected behavior:

Returned organizations from Enterprise query

Please note that I copied the query into GitHub GraphQL Explorer and it returns the organizations as expected.

{
  "data": {
    "viewer": {
      "login": "djthornton1212"
    },
    "rateLimit": {
      "limit": 5000,
      "cost": 1,
      "remaining": 4989,
      "resetAt": "2022-04-07T19:23:33Z",
      "used": 11
    },
    "enterprise": {
      "organizations": {
        "nodes": [
          {
            "login": "<1st org>"
          },
          {
            "login": "<2nd org>"
          }
        ],
        "totalCount": 2,
        "pageInfo": {
          "hasNextPage": false,
          "endCursor": "Y3Vyc29yOnYyOpKuQ2FyZWdpbGl0eS1GZWTOBdckCA=="
        }
      }
    }
  }
}

We are trying to run this tool from a GitHub runner (macOS), on a schedule.
Currently this doesn't work completely yet because the git commands that are used to commit the codeql file rely on local ssh configuration or user prompts. We would like it if this part could also make use of the PAT / GitHub App auth.

I have tested a change for this locally which seems to work and added commands for wsl/linux, . I still need to figure out how to ensure that the token will not be shown in the output of the script. (currently it prints all the commands showing the secret).

Do you think this can be added? @NickLiffen
If so I will clean it up a bit and create a pull request for you to review/refactor.
Also do you think it should always commit the file through the app auth, or should it be an option in the .env file ?

Kind regards,
Jors

@NickLiffen
The commands and file structures don't work so well on windows, for example, using "/" on windows gives Error: spawn C:\Windows\system32\cmd.exe ENOENT, improvement was to detect the platform in which the user works on and use "" if it's a window or "/" if it's a mac.

• using a Desktop as a destination on a windows machine can be ambiguous for some organizations like EY, as there are several users on the enterprise machine, so to be specific, we recommend using Documents as the destination as the user will most of the time have direct access to the Documents directory. it was giving us in EY the same spawn error above.

• { command: "mkdir -p .github/workflows" } was giving syntax error too, at least on windows, So I enhance it by explicitly specifying .github/workflows as string, something like this. mkdir -p ".github/workflows"

• _The two first enhancements have taken care of the spawn error in the windows machine. and the third one got rid of the syntax error. _