cloudquery / cq-provider-k8s

CloudQuery Provider for Kubernetes

This moved to cloudquery monorepo https://github.com/cloudquery/cloudquery

CloudQuery K8S Provider

This CloudQuery provider transforms K8S resources to a relational database.

What is CloudQuery

The open-source cloud asset inventory powered by SQL.

CloudQuery extracts, transforms, and loads your cloud assets into normalized PostgreSQL tables. CloudQuery enables you to assess, audit, and evaluate the configurations of your cloud assets.

Links

• Homepage: https://cloudquery.io
• Documentation: https://docs.cloudquery.io
• CloudQuery Hub (providers & policies documentation): https://hub.cloudquery.io/
• Discord: https://cloudquery.io/discord

cq-provider-k8s's Issues

Panic when trying to fetch from a specifc context

Describe the Bug

Setting contexts parameter in config.hcl causees panic.

Expected Behavior

cloudquery should fetch from this context.

CloudQuery Version

Version: 0.22.7
Commit: fb5a1447e0851d96af0fb7d522585bf78566ff3f
Date: 2022-04-19T08:55:30Z

cq-provider-k8s version: v0.4.2

Debug Output

See attached file.
log.txt

Steps to Reproduce

config.hcl file:

// Configuration AutoGenerated by CloudQuery CLI
cloudquery {
  plugin_directory = "./cq/providers"
  policy_directory = "./cq/policies"

  provider "k8s" {
    version = "latest"
  }

  connection {
    dsn = "postgres://postgres:pass@localhost:5432/postgres?sslmode=disable"
  }
}

// All Provider Configurations

provider "k8s" {
  configuration {
    // Optional. Set contexts that you want to fetch. If it is not given then all contexts from config are iterated over.
    contexts = ["one"]
  }
  // list of resources to fetch
  resources = [
    "apps.daemon_sets",
    "apps.deployments",
    "apps.replica_sets",
    "apps.stateful_sets",
    "batch.cron_jobs",
    "batch.jobs",
    "core.endpoints",
    "core.limit_ranges",
    "core.namespaces",
    "core.nodes",
    "core.pods",
    "core.resource_quotas",
    "core.service_accounts",
    "core.services",
    "networking.network_policies",
    "rbac.role_bindings",
    "rbac.roles"
  ]
  // enables partial fetching, allowing for any failures to not stop full resource pull
  enable_partial_fetch = true
}

cloudquery fetch

Additional Context

References

See also #70

Add k8s Contribution Guide

New Resource

Use Case

Additional context

update docs

Support for querying labels and annotations

It would be tremendously useful to include labels and annotations as something that could be queried. I might have time to make a PR for this at some point. Would you prefer a single polymorphic labels table that is shared across resources, or something like PodLabels?

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (github>cloudquery/.github//.github/renovate-go-default.json5)

Fix integration tests

Describe the Bug

Expected Behavior

CloudQuery Version

Debug Output

Steps to Reproduce

Additional Context

References

Add nightly fetch workflow to k8s

Add nightly fetch workflow to k8s

update goreleaser to support all platforms

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

This repository currently has no open or pending branches.

Detected dependencies

github-actions

.github/workflows/issue_to_project.yml

• leonsteinhaeuser/project-beta-automations v1.2.1

.github/workflows/lint_doc.yml

• actions/checkout v3
• dorny/paths-filter v2
• actions/setup-go v3
• actions/cache v3

.github/workflows/lint_golang.yml

• actions/checkout v3
• dorny/paths-filter v2
• actions/setup-go v3
• golangci/golangci-lint-action v3

.github/workflows/pr_title.yml

• amannn/action-semantic-pull-request v3.4.6

.github/workflows/release-pr.yml

• google-github-actions/release-please-action v3
• tubone24/update_release undefined

.github/workflows/release.yml

• booxmedialtd/ws-action-parse-semver undefined
• actions/checkout v2
• actions/setup-go v2
• goreleaser/goreleaser-action v2
• crazy-max/ghaction-import-gpg v3
• goreleaser/goreleaser-action v2

.github/workflows/test_policy.yml

• actions/checkout v2
• actions/setup-go v3
• actions/cache v3

.github/workflows/test_unit.yml

• actions/checkout v3
• dorny/paths-filter v2
• actions/setup-go v3
• actions/cache v3

.github/workflows/validate_release.yml

• actions/checkout v3
• actions/setup-go v3
• goreleaser/goreleaser-action v2

gomod

go.mod

• github.com/cloudquery/cq-provider-sdk v0.9.5
• github.com/cloudquery/faker/v3 v3.7.5
• github.com/golang/mock v1.6.0
• github.com/hashicorp/go-hclog v1.0.0
• github.com/thoas/go-funk v0.9.1
• k8s.io/api v0.22.1
• k8s.io/apimachinery v0.22.1
• k8s.io/client-go v0.22.1

terraform

resources/integration_tests/infra/provider.tf

• kubernetes undefined

resources/integration_tests/infra/terraform.tf

• terraform ~> 2.6

---

• Check this box to trigger a request for Renovate to run again on this repository

Very weird behaviour when fetching from different clusters/contexts

Describe the Bug

Two separate but related issues here:

1. Our documentation states that cloudquery fetched from "all available kubernetes contexts in the config file". However, cloudquery only actually fetches from the current-context in the kubeconfig file.
2. cloudquery doesn't delete old entries if the user switched contexts. This may be related to "purge".

Expected Behavior

1. only fetching from current context makes sense to me, so I think just changing docs is OK. I actually think fetching from all contexts is a bad idea, because kubeconfig files tend to accumulate a lot of stale contexts (e.g. mine has a bunch of EKS clusters from AWS accounts that no longer exist).
2. I definitely expect old entries to be deleted if a user switched context. Should the user really explicitly "purge" the old records? I don't love that user experience.

CloudQuery Version

Version: 0.22.7
Commit: fb5a1447e0851d96af0fb7d522585bf78566ff3f
Date: 2022-04-19T08:55:30Z

cq-provider-k8s version: v0.4.2

Steps to Reproduce

Assume kubectl and minikube is installed.

minikube start --profile one
minikube start --profile two
minikube start --profile three

kubectl config use-context one
cloudquery fetch
# k8s_core_nodes now contains 1 row, for profile 'one'
kubectl config use-context two
cloduquery fetch
# k8s_core_nodes now contains 2 rows - node 'two', and node 'one' that wasn't deleted. 
kubectl config use-context three
cloudquery fetch
# k8s_core_nodes now contains 3 rows - node 'three', and nodes 'one' and 'two' that weren't deleted.

Add error classifier

Add Support for k8s error classifier to classify access issues

create e2e testing for k8s

Check api availability before doing any calls to k8s server

K8s server allows to get api schema by calling kClient.OpenAPISchema()
The method returns list of paths available on the server.
The data from response can be used to avoid calls to endpoints that are not supported bu current version of k8s server

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• fix(deps): Update tubone24/update_release digest to cf5b1c5
• fix(deps): Update leonsteinhaeuser/project-beta-automations action to v1.3.0
• fix(deps): Update module go to 1.19

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• fix(deps): Update module k8s.io/client-go to v12

Detected dependencies

github-actions

.github/workflows/issue_to_project.yml

• leonsteinhaeuser/project-beta-automations v1.2.1

.github/workflows/lint_doc.yml

• actions/checkout v3
• dorny/paths-filter v2
• actions/setup-go v3
• actions/cache v3

.github/workflows/lint_golang.yml

• actions/checkout v3
• dorny/paths-filter v2
• actions/setup-go v3
• golangci/golangci-lint-action v3

.github/workflows/pr_title.yml

• amannn/action-semantic-pull-request v4

.github/workflows/release-pr.yml

• google-github-actions/release-please-action v3
• tubone24/update_release 2146f1550a23d883b8ea0c036298ed74cd65eac6

.github/workflows/release.yml

• booxmedialtd/ws-action-parse-semver 966a26512c94239a00aa10b1b0c196906f7e1909
• actions/checkout v3
• actions/setup-go v3
• goreleaser/goreleaser-action v3
• crazy-max/ghaction-import-gpg v5
• goreleaser/goreleaser-action v3

.github/workflows/test_policy_sql.yml

• actions/checkout v3
• actions/setup-go v3
• actions/cache v3
• undefined no version found

.github/workflows/test_unit.yml

• actions/checkout v3
• dorny/paths-filter v2
• actions/setup-go v3
• actions/cache v3
• undefined no version found

.github/workflows/validate_release.yml

• actions/checkout v3
• actions/setup-go v3
• goreleaser/goreleaser-action v3

gomod

go.mod

• go 1.18
• github.com/cloudquery/cq-provider-sdk v0.14.7
• github.com/cloudquery/faker/v3 v3.7.7
• github.com/golang/mock v1.6.0
• github.com/hashicorp/go-hclog v1.2.2
• github.com/thoas/go-funk v0.9.2
• k8s.io/api v0.24.3
• k8s.io/apimachinery v0.24.3
• k8s.io/client-go v0.24.3

terraform

resources/integration_tests/infra/k8s_apps_daemon_sets.tf

• nginx 1.23.1

resources/integration_tests/infra/k8s_apps_deployments.tf

• nginx 1.23.1

resources/integration_tests/infra/k8s_apps_replica_sets.tf

• nginx 1.23.1

resources/integration_tests/infra/k8s_apps_stateful_sets.tf

• mongo bionic

resources/integration_tests/infra/k8s_batch_cron_jobs.tf

• busybox no version found

resources/integration_tests/infra/k8s_batch_jobs.tf

• perl no version found

resources/integration_tests/infra/k8s_core_endpoints.tf

• undefined no version found

resources/integration_tests/infra/k8s_core_limit_ranges.tf

• undefined no version found
• undefined no version found

resources/integration_tests/infra/k8s_core_namespaces.tf

• undefined no version found

resources/integration_tests/infra/k8s_core_nodes.tf

• nginx 1.23.1

resources/integration_tests/infra/k8s_core_pods.tf

• nginx 1.23.1

resources/integration_tests/infra/k8s_core_resource_quotas.tf

• undefined no version found
• undefined no version found

resources/integration_tests/infra/k8s_core_service_accounts.tf

• undefined no version found
• undefined no version found

resources/integration_tests/infra/k8s_core_services.tf

• undefined no version found
• nginx 1.23.1

resources/integration_tests/infra/k8s_networking_network_policies.tf

• undefined no version found

resources/integration_tests/infra/k8s_rbac_role_bindings.tf

• undefined no version found

resources/integration_tests/infra/k8s_rbac_roles.tf

• undefined no version found

resources/integration_tests/infra/provider.tf

• kubernetes no version found

resources/integration_tests/infra/terraform.tf

• terraform ~> 2.6

add changelog

k8s provider README is very lacking.

specifically authentication section - you don't actually need a KUBECONFIG environment vairable - cloudquery will just use the default if one isn't sepecified...