cloudquery / .github

The open-source cloud asset inventory powered by SQL.

CloudQuery .github repository

This repository contains templates and renovate configurations that are used across all CloudQuery repositories.

Templates

• Issue template
• Code of conduct
• Security policy

Renovate configurations

We use renovate for dependency updates via a GitHub Action. As renovate runs in the context of this repository, it uses a self hosted renovate configuration file to generate dependencies updates. We have a few base configurations that repositories extend from.

.github's Issues

Change the endpoints workflow to commit changes instead of verifying if changes were made

I noticed we have a CI workflow to detect changes to a JSON file

.github/workflows/repo-specific/cq-provider-aws/endpoints.yml

Line 25 in 1719777

test "$(git status -s ./client/data/partition_service_region.json | wc -l)" -eq 0

Instead of detecting changes and failing the CI, we could auto commit the changes when the PR that generated those was opened.

For example, instead of:

on:
  push:
    branches:
      - main
      
.....

      - name: Fail if file is changed
              run: |
                test "$(git status -s ./client/data/partition_service_region.json | wc -l)" -eq 0

We could:

on:
  pull_request:
    branches:
      - main
      
.....
      - uses: actions/checkout@v3
        with:
          # A personal access token is required so to re-run the CI. See here https://github.com/stefanzweifel/git-auto-commit-action#commits-made-by-this-action-do-not-trigger-new-workflow-runs
          token: ${{ secrets.GH_CQ_BOT }}
      - uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: fix: Sync endpoints

Happy to make the change if this makes sense to everyone

Test

test issue

policy workflow to use main branch

Describe the bug

providers policy test validation, only runs on latest version, if there is a breaking change we require to tag before those tests can pass, moreover we might miss bugs in latest.

Expected Behavior

pull latest core version instead of latest tag

Steps to Reproduce

Possible Solution

No response

Provider and CloudQuery version

main

Additional Context

No response

chore(release): Add contributors to release changelog

With the switch to release-please the release change log is missing the contributors list.

We should find a way to add it.

Looks like https://github.com/googleapis/release-please/blob/f384c7581e1aa562a7a015d2039ee463e63601e2/docs/customizing.md#changelog-types is the configuration we need

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (github>cloudquery/.github//.github/renovate-default.json5)

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

This repository currently has no open or pending branches.

Detected dependencies

github-actions

.github/workflows/Issue_to_project.yml

• leonsteinhaeuser/project-beta-automations v1.2.1

.github/workflows/sync_files.yml

• actions/checkout main
• cloudquery/repo-file-sync-action feat/templating_v2

.github/workflows/sync_repo_settings.yml

• actions/checkout v3
• actions/setup-node v3
• actions/checkout v3

---

• Check this box to trigger a request for Renovate to run again on this repository

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

github-actions

.github/workflows/renovate.yml

• actions/checkout v4
• renovatebot/github-action v40.1.11

.github/workflows/sync-labels.yml

• actions/checkout v4
• micnncim/action-label-syncer v1

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (github>cloudquery/.github//.github/renovate-go-default.json5)

Our sync action fails to create directories when syncing single files

See https://github.com/cloudquery/.github/runs/6288442440?check_suite_focus=true#step:3:217

chore(deps): Improve renovate schedule time

Once we release a new SDK version, it takes a bit of time for renovate to open the dependency PRs.

I think the renovate GitHub app runs it on an hourly non configurable basis and we can trigger manually via the dependency dashboard issue.

We could use https://github.com/renovatebot/github-action/tree/c33d138d871e6f9222734c3cfda81c513b5b8bd6#example if the GitHub app schedule is not configurable

Make use of GitHub Action From marketplace

Rather than manually downloading (curl ... or wget ...) the binary we should use the github.com/cloudquery/setup-cloudquery action

Docs: Add README to this repository

We should document all the various automations that this repo has.

feat: Use a GitHub app instead of a bot account for elevated tokens

Context

We use a GitHub personal access token from a bot (cq-bot) account when we need to run GitHub actions with elevated permissions.
There are several reasons not to use the built-in GitHub Actions GITHUB_TOKEN:

1. Workflows using the built-in GITHUB_TOKEN can't trigger new workflow runs. See here. This means that if we create release PRs, renovate PRs, etc. they won't trigger any tests or other CI workflows (blocking them from getting merged due to branch protection).
2. The built-in GITHUB_TOKEN is scoped to the repo it runs in, so we can't use it for cross repo automation (e.g. open PRs in other repos from this repo).
3. When executed from forked repositories, the built-in GITHUB_TOKEN has only read permissions. We use the pull_request_target event to handle such cases, see

   .github/workflows/providers/test_integration.yml

   Line 10 in 62af2e4

   pull_request_target:

   .

Problem

Using a bot account works well for our use cases, however it has a few downsides:

1. We might hit GitHub rate limits if we have more automation (limit is 5000 requests per hour)
2. Permissions for personal access tokens are not very granular
3. The bot user account takes up a seat in our GitHub organization and we need to save its credentials. See here
4. Personal access tokens are long living tokens which pose a security risk

TLDR: The main issue I've experienced is with rate limits, as the limits are per account and not per token.

Suggested solution

We can create a GitHub app and use it for generating tokens. See here.

Advantages:

1. Better rate limits for GitHub apps
2. More granular permissions for GitHub apps (can be installed on specific repos too)
3. GitHub app generated tokens expire after 1 hour
4. GitHub apps don't take up a seat

Bug(github-action-sync-files): the sync files action fails to complete

Describe the bug

We use a GitHub action to sync various files and template from this repo to other repos.

See https://github.com/cloudquery/.github/blob/171977710ab1e8d6db4aa0ccbdcb2372e4101e58/.github/workflows/sync_files.yml and https://github.com/cloudquery/.github/blob/171977710ab1e8d6db4aa0ccbdcb2372e4101e58/.github/sync.yml

The action fails at the moment. It seems one reason is not being able to overwrite existing PRs, see https://github.com/cloudquery/.github/runs/6053121350?check_suite_focus=true#step:3:1247

Additionally there are a bunch of warnings when running the action about not being able to copy files, see https://github.com/cloudquery/.github/runs/6053121350?check_suite_focus=true#step:3:1023

Expected Behavior

The GitHub action should sync files should complete successfully and sync files based on the config file

Steps to Reproduce

Merge any commit to main to trigger the action, or re-run https://github.com/cloudquery/.github/actions/runs/2179547340

Possible Solution

For the failure to copy warnings (without further debugging) we might need to ensure the destination directory structure exists here https://github.com/cloudquery/repo-file-sync-action/blob/94d624a88ce5d3cc11f61eb87e5384846f6d78f0/src/helpers.js#L94.

For the errors to create PRs we might need to handle existing PRs here https://github.com/cloudquery/repo-file-sync-action/blob/94d624a88ce5d3cc11f61eb87e5384846f6d78f0/src/git.js#L183.

Another alternative is to stop using our fork of the action (the original handles existing PRs), though I don't have enough context to know the reason we have our own fork

Provider and CloudQuery version

N/A

Additional Context

No response

Add support for gofmt in golang-ci

Describe the bug

Need to ensure the build tags are consistent and up to date

Expected Behavior

use latest version of tags

Steps to Reproduce

na

Possible Solution

No response

Provider and CloudQuery version

any

Additional Context

No response

Enable Removal of Files in Target Repos

Right now we can add files to repos, but to remove files we need to go to that repo to remove them.

Current solution enables removal of orphaned files only if syncing an entire directory. We cannot use this at this time because we are syncing multiple source directories to a single directory

Drop Support for PG10

Move minimum version to 11.x in all tests

.golangci.yml dec-order controverses cq-gen generation order

Describe the bug

cq-gen generates user defined funcitons and types under User Defined Helpers at the end of the file.
dec-order in linter configuration required types to be above functions

Expected Behavior

cq-gen should generate code acording to dec-order bloc or dec-order should be disabled.

Steps to Reproduce

generate file with user type and funtion and run golangci-lint

Possible Solution

No response

Provider and CloudQuery version

Additional Context

No response