suse / stratos-metrics

Helm chart for deploying Prometheus and the Cloud Foundry Firehose exporter for use in Stratos

Stratos Metrics

Stratos Metrics provides a Helm chart for deploying Prometheus and the following metrics exporters to Kubernetes:

• Cloud Foundry Firehose Exporter (enabled by default)
• Cloud Foundry Exporter (disabled by default)
• Kubernetes State Metrics Exporter (disabled by default)

The Stratos Metrics Helm Chart deploys a Prometheus server and the configured Exporters and fronts the Prometheus server with an nginx server to provide authenticated access to Prometheus (currently basic authentication over https).

When required by configuration, it also contains an initialization script that will setup users in the UAA that have correct scopes/permissions to be able to read data from the Cloud Foundry Firehose and/or API.

Lastly, the Helm chart generates a small metadata file in the root of the nginx server which is used by Stratos to determine which Cloud Foundry and Kubernetes clusters the Prometheus server is providing Metrics for.

Installation

The Helm chart is published to the Stratos Helm repository. Ensure you've followed the Stratos requirements for deploying to Kubernetes - https://github.com/SUSE/stratos/tree/master/deploy/kubernetes#requirements.

You will need to have the SUSE Helm repository added to your Helm setup, if you do not, run:

helm repo add suse https://kubernetes-charts.suse.com/

You may need to refresh from this repository, if you already had it added, by running:

helm repo update

You should see the metrics chart is available when running:

helm search metrics -l

You can install with:

helm install suse/metrics --namespace=metrics -f <CONFIG_VALUES>.yaml

Where <CONFIG_VALUES>.yaml is the name of a configuration values file that you must create (see below).

If you want to use a Load Balancer or change the external IP, see the Advanced Topics section below.

Note, if you are using SUSE Cloud Foundry, you can use your scf_config_values.yaml file that you used when deploying it.

This will create a Stratos Metrics instance in a namespace called metrics in your Kubernetes cluster.

Connecting Metrics to Stratos

Once deployed, the Prometheus server should be accessible via https, ready to connect to Stratos.

When connecting, the default credentials are:

Username: metrics
Password: s3cr3t

NOTE: We recommend that you do not use the defaults - they can be changed via the following helm chart values:

metrics:
  username: <USERNAME>
  password: <PASSWORD>

Exporter Configuration

Exporters will connect to their targets and export metrics into Stratos's Prometheus.

Cloud Foundry Firehose Exporter

The Cloud Foundry Firehose Exporter provides data required by Stratos to show CF Application and Diego Cell information. For more information about the type of data collected see here.

This exporter can be enabled/disabled via the Helm value firehoseExporter.enabled. By default this exporter is enabled.

You must provide the following Helm Chart values for this Exporter to work correctly:

• cloudFoundry.apiEndpoint - API Endpoint of the Cloud Foundry API Server
• cloudFoundry.uaaAdminClient - Admin client of the UAA used by the Cloud Foundry server
• cloudFoundry.uaaAdminClientSecret - Admin client secret of the UAA used by the Cloud Foundry serve
• cloudFoundry.skipSslVerification - Whether to skip SSL verification when communicating with Cloud Foundry and the UAA APIs

You can scale the firehose nozzle in Stratos-Metrics by specifying the following override:

firehoseExporter:
  instances: 1

Please note, the number of firehose nozzles should be proportional to the number of Traffic Controllers in your Cloud Foundry (see docs). Otherwise, Loggregator will not split the firehose between the nozzles.

Cloud Foundry Exporter

The Cloud Foundry Exporter is not required to view metrics in Stratos but can provide interesting data for analysis separately. The type of data collected revolves around CF resource types such as organizations, spaces, service bindings, stacks, etc. For more information about the type of data collected see here.

This exporter can be enabled/disabled via the Helm value cfExporter.enabled. By default this exporter is disabled.

You must provide the following Helm Chart values for this Exporter to work correctly:

• cloudFoundry.apiEndpoint - API Endpoint of the Cloud Foundry API Server
• cloudFoundry.uaaAdminClient - Admin client of the UAA used by the Cloud Foundry server
• cloudFoundry.uaaAdminClientSecret - Admin client secret of the UAA used by the Cloud Foundry serve
• cloudFoundry.skipSslVerification - Whether to skip SSL verification when communicating with Cloud Foundry and the UAA APIs

Kubernetes State Metrics Exporter

The Kubernetes State Metrics Exporter can be enabled/disabled via the Helm value prometheus.kubeStateMetrics.enabled. By default this exporter is disabled.

You must provide the following Helm Chart values for this Exporter to work correctly:

• kubernetes.apiEndpoint - The API Endpoint of the Kubernetes API Server

Node Exporter

The Node Exporter can be enabled/disabled via the Helm value prometheus.nodeExporter.enabled. By default this exporter is disabled.

Note: If your cluster is deployed with RBAC you must set the Helm value prometheus.podSecurityPolicy.enabled=true.

Helm Chart Configuration

The following table lists the configurable parameters of the Metrics chart and their default values.

Parameter	Description	Default
imagePullPolicy	Image pull policy	IfNotPresent
metrics.username	Basic Auth username for accessing metrics services	metrics
metrics.password	Basic Auth password for accessing metrics services_see values.yaml_
metrics.service.type	Type of the metrics service to create	ClusterIP
metrics.service.annotations	Annotations for the metrics service	[]
metrics.service.externalIPs	External IP addresses for the metrics service	[]
metrics.service.loadBalancerIP	IP address to assign to the load balancer for the metrics service (if supported)
metrics.service.loadBalancerSourceRanges	List of IP CIDRs allowed access to load balancer (if supported)	[]
metrics.service.servicePort	Service port for the metrics service	443
metrics.service.nodePort	Node port for the metrics service (ignored if metrics.service.type is not NodePort)
metrics.service.externalName	External name for the metrics service
metrics.service.ingress.enabled	Enable ingress for the metrics service	false
metrics.service.ingress.host	Host for the ingress resource
metrics.service.ingress.secretName	Name of an existing secret containing the TLS certificate for ingress
metrics.service.ingress.annotations	Annotations to be added to the ingress resource
metrics.service.ingress.extraLabels	Additional labels to be added to the ingress resource
nginx.ssl.cert	TLS Certificate for the metrics service	self-signed dev certificate
nginx.ssl.key	TLS Private Key for the metrics service	self-signed dev certificate
cloudFoundry.apiEndpoint	API Endpoint of the Cloud Foundry API Server (required by the Firehose and CF Exporters)
cloudFoundry.uaaAdminClient	Admin client of the UAA used by the Cloud Foundry server (required by the Firehose and CF Exporters)	admin
cloudFoundry.uaaAdminClientSecret	Admin client secret of the UAA used by the Cloud Foundry server (required by the Firehose and CF Exporters)
cloudFoundry.skipSslVerification	Whether to skip SSL verification when communicating with Cloud Foundry and the UAA APIs	"true"
firehoseExporter.enabled	Flag to enable ot disable the Prometheus Firehose Exporter	true
firehoseExporter.instances	Number of instance of the firehose exporter to scale to	1
firehoseExporter.dopplerUrl	URL of the Cloud Foundry Doppler endpoint to monitor (used by the firehose exporter and takes precedence over specifying the Cloud Foundry API endpoint)
cfExporter.enabled	Flag to enable ot disable the Prometheus CF Exporter	false
kubernetes.apiEndpoint	URL of the Kubernetes API Server
prometheus.kubeStateMetrics.enabled	Enables the Kubernetes state metrics prometheus Exporter	false
prometheus.nodeExporter.enabled	Enables the Node Exporter	false
prometheus.podSecurityPolicy.enabled	Set to true if the Kubernetes cluster supports Role-based access control	false
kube.auth	Set to "rbac" if the Kubernetes cluster supports Role-based access control	"rbac"
prometheus.server.persistentVolume.storageClass	Storage class to use for the Prometheus server	(use default storage class)
kube.clusterDomain	Kubernetes domain	cluster.local
metrics.pspEnabled	Enable Pod Security Policies. Set this to true if you cluster is configured with PSPs enabled	false
metrics.pspName	Name of an existing Pod Security Policy to use instead of the one created by the chart when PSPs are enabled
metrics.pspAnnotations	Annotations to be added to all pod security policy resources
metrics.pspExtraLabels	Additional labels to be added to all pod security policy resources

Advanced Topics

Deploying with Pod Security Policies enabled

If your Kubernetes cluster has Pod Security Policies enabeld, you need to supply additional Helm values:

metrics:
  pspEnabled: true
prometheus
  podSecurityPolicy:
    enabled: true

Deploying to a cluster with a configured Kubernetes Domain

The default domain used by a Kubernetes cluster is cluster.local. If your cluster has been configured with a different domain name, you must supply this in the Helm value kube.clusterDomain.

Using a Load Balancer

If your Kubernetes deployment supports automatic configuration of a load balancer (e.g. Google Container Engine), specify the parameters metrics.service.type=LoadBalancer when installing.

helm install stratos/metrics --devel --namespace=metrics -f <CONFIG_VALUES>.yaml --set metrics.service.type=LoadBalancer

Specifying an External IP

If the Kubernetes cluster supports external IPs for services (see Service External IPs), then the following argument can be provided:

helm install stratos/metrics --devel --namespace=metrics -f <CONFIG_VALUES>.yaml --set metrics.service.externalIP=192.168.100.100

Using an Ingress Controller

If your Kubernetes Cluster supports Ingress, you can expose Stratos Metrics through Ingress by supplying the appropriate ingress configuration when installing.

This configuration is described below:

Parameter	Description	Default
metrics.service.ingress.enabled	Enables ingress	false
metrics.service.ingress.annotations	Annotations to be added to the ingress resource.	{}
metrics.service.ingress.extraLabels	Additional labels to be added to the ingress resource.	{}
metrics.service.ingress.host	The host name that will be used for the Stratos Metrics service.
metrics.service.ingress.secretName	The existing TLS secret that contains the certificate for ingress.

You must provide metrics.service.ingress.host when enabling ingress.

By default a certificate will be generated for TLS. You can provide your own certificate by creating a secret and specifying this with metrics.service.ingress.secretName.

Note: If you do not supply metrics.service.ingress.host but do supply env.DOMAIN then the host metrics.[env.DOMAIN] will be used.

Deploying Metrics from a Private Image Repository

If the images used by the chart are hosted in a private repository, the following needs to be specified. Save the following to a file called private_overrides.yaml. Replace REGISTRY USER PASSSWORD, REGISTRY USERNAME, REGISTRY URL with the appropriate values. USER EMAIL can be left blank.

Note: Stratos Metrics uses the Prometheus Helm Chart as a sub-chart - hence the secret for the private registry must be specified separately for these two components.

prometheus:
  imagePullSecrets:
  - name: regsecret

kube:
  registry:
    password: <REGISTRY USER PASSWORD>
    username: <REGISTRY USERNAME>
    hostname: <REGISTRY URL>
    email: <USER EMAIL or leave blank>

To deploy stratos/metrics helm chart:

helm install stratos/metrics -f private_overrides.yaml --namespace=metrics

Advanced Prometheus Configuration

Stratos Metrics uses the Prometheus Helm chart (https://github.com/helm/charts/tree/master/stable/prometheus) as a sub-chart.

You can override settings for Prometheus, as described in this sub-chart, but prefixing the value with prometheus. For example, prometheus.server.persistentVolume.storageClass changes the storage class used by the Prometheus server.

stratos-metrics's Issues

Bump Kube State Metrics Exporter to version 1.9.7

Update to latest firehose exporter

Helm chart should perform some basic checks and fail if values missing

Currently you can deploy with firehose exporter enabled and without the required config and the chart will deploy but the init job will fail.

We should do checks in the helm chart templates and fail early if the required values are not supplied.

Add support for ingress

You can now deploy metrics with ingress - see here: https://github.com/SUSE/stratos-metrics/blob/master/README.md#using-an-ingress-controller

Bump Prometheus version to 2.20.1 and Helm Chart Version 11.15.0

Default metrics self-signed certificate is out of date

Setting nginx credentials failed, remained default

I deployed metrics to CaaSP3 as well as to Azure AKS - in both cases the nginx credentials remained default:

> kubectl logs stratos-metrics-cfdevio-fcdc877bb-55mkm -n metrics
Checking if certificate has been supplied!
TLS certificate detected continuing, starting nginx.
Create username/password for authentication
Adding password for user metrics
Starting nginx

while my metrics-config-values.yaml contained (e.g. azure):

secrets:
  CLUSTER_ADMIN_PASSWORD: password
  UAA_ADMIN_CLIENT_SECRET: password
env:
  DOMAIN: 40.68.160.43.omg.howdoi.website
  UAA_HOST: uaa.40.68.160.43.omg.howdoi.website
  UAA_PORT: 2793
  GARDEN_ROOTFS_DRIVER: "overlay-xfs"
kube:
  external_ips: ["10.240.0.6", "10.240.0.5", "10.240.0.4"]
  storage_class:
    persistent: "default"
    shared: "shared"
  registry:
    hostname: "registry.suse.com"
    username: ""
    password: ""
  organization: "cap"
  auth: "rbac"
  psp:
    nonprivileged: "suse.cap.psp"
    privileged: "suse.cap.psp"
---
kubernetes:
  authEndpoint: https://svollath-svollath-cap-aks-f416cc-f0ebd67d.hcp.westeurope.azmk8s.io:443
prometheus:
  kubeStateMetrics:
    enabled: true
  imagePullSecrets:
  - name: regsecret
nginx:
  username: admin
  password: password

helm install suse/metrics --name metrics --namespace metrics -f metrics-config-values.yaml

Update Prometheus (2.19 or 2.20?)

Lots of fixes and improvements in upstream Prometheus. A customer has asked for an update.

https://github.com/prometheus/prometheus/releases

Migrate from `extensions/v1beta` to `apps/v1/ for kubernetes 1.9+

Nginx pod crash loop backoff when domain name is not cluster.local

If your kube cluster does not use the default domain name of cluser.local then metrics will fail to deploy - you'l get crash loop backoffs.

You can use minikube to reproduce, by starting with the flag --dns-domain, e.g. '--dns-domain='cluster.demo'.

Bump Prometheus Node Exporter to version 1.0.1

Bump CF Firehose exporter to version 6.1.0

Prometheus Subchart missing helm recommended labels

Firehose exporter can sometimes stop ingesting metrics

Clear text username/pwds are being displayed when describing a stratos metrics pod

Deploying Metrics on AKS is failing

Trying to deploy Metrics on AKS is giving validation error:

helm install suse/metrics --name susecf-metrics --namespace metrics --values ../../scf-config-values.yaml --values ../../stratos-metrics-values.yaml --debug
[debug] Created tunnel using local port: '41083'

[debug] SERVER: "127.0.0.1:41083"

[debug] Original chart version: ""
[debug] Fetched ...

[debug] CHART PATH: /home/.../.helm/cache/archive/metrics-1.1.0.tgz

Error: validation failed: error validating "": error validating data: ValidationError(Job.spec.template.spec): unknown field "backoffLimit" in io.k8s.api.core.v1.PodSpec

The only way I was able to make it work is by hashing the backofflimit.

I am using K8s version:"1.14.8"

Bump CF Exporter to version 0.10.0

Bump base image OS to newer Leap15_1/SLE 15 SP1

To verify:

deploy metrics using helm
use kubectl to exec into one of the containers
cat /etc/os-release to validate that the newer OS image is being used

kube-state-metrics Failed to list *v1beta1.StatefulSet

Not seeing any Kubernetes metrics with v1.16.2

2020-03-07T00:12:27.187743494Z E0307 00:12:27.187514 1 reflector.go:205] k8s.io/kube-state-metrics/collectors/collectors.go:62: Failed to list *v1beta1.Deployment: the server could not find the requested resource
2020-03-07T00:12:27.187819769Z E0307 00:12:27.187793 1 reflector.go:205] k8s.io/kube-state-metrics/collectors/collectors.go:62: Failed to list *v1beta1.StatefulSet: the server could not find the requested resource

I notice Rancher had a similar issue:
rancher/rancher#23556

I think an upgrade of kube-state-metrics should solve this. Latest release is:
https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.5

To reproduce, deploy metrics and view the log of the kube state metrics exporter - you'll see error messages like those above.

Also, if you go into the Prometheus UI, you'll see that there are no metrics of the name container_*.

Add bosh-prometheus/cf_exporter

The Cloud Foundry Prometheus Exporter scrapes information from the Cloud Foundry API which can be used to by admins to see usage information over time (e.g. for creating billing reports).

IIUC, adding this to stratos-metrics would allow admins to use Grafana dashboards such as these: https://grafana.com/orgs/sprignernature ... and enable similar views within Stratos itself.

Implementing rules and alerts are not possible

Description:
Is it currently nos possible during the creation process of stratos-metrics to implement alerts and rules.

Expectation
As service provider of PaaS platform on NoOps envs, I need to be able to setting up alerts and rules during the creation process of an instance of stratos metrics. It could be more suitable if we can add it using custom parameters in the charts.

Br