vmware-archive / wavefront-adapter-for-istio

Wavefront by VMware Adapter for Istio is an adapter for Istio to publish metrics to Wavefront by VMware.

Wavefront by VMware Adapter for Istio

Wavefront by VMware Adapter for Istio is an adapter for Istio to publish metrics to Wavefront by VMware.

Note: The master branch is used for active development and can become unstable. Please refer to the Quick Start from version 0.1.5 to install a stable version of the adapter.

Quick Start

This adapter could be installed either via Helm or via the standard method.

Helm Installation

Helm is the preferred way of installing this adapter. Please see the Helm Hub to learn to install this adapter using Helm.

Standard Installation

Prerequisites

To deploy this adapter, you will need a cluster with the following setup.

• Kubernetes v1.15+
• Istio v1.4 or v1.5 or v1.6 or v1.7

Note: From Istio v1.5.x onwards Mixer is disabled by default. Enable Mixer with the following step:

Istio v1.5.x

istioctl manifest apply --set values.telemetry.v1.enabled=true --set values.telemetry.v2.enabled=false --set components.telemetry.enabled=true --set components.citadel.enabled=true

Istio v1.6.x or v1.7.x

istioctl install --set values.telemetry.v1.enabled=true --set values.telemetry.v2.enabled=false --set components.telemetry.enabled=true

# Use below command if Istio is deployed in non default namespace, replace <NAMESPACE> with namespace name into which Istio is deployed.
istioctl install --set values.telemetry.v1.enabled=true --set values.telemetry.v2.enabled=false --set components.telemetry.enabled=true --set values.global.istioNamespace=<NAMESPACE>

Configuration

1. Download the configuration.

$ curl -LO https://raw.githubusercontent.com/vmware/wavefront-adapter-for-istio/0.1.5/install/config.yaml

2. If you want the metrics to be published to the Wavefront instance directly, supply the direct params for the wavefront-handler like so:

params:
  direct:
    server: https://YOUR-INSTANCE.wavefront.com
    token: YOUR-API-TOKEN

Instructions for generating an API token can be found in the Wavefront by VMware docs.

If you want the metrics to be published to the Wavefront Proxy instead, supply the proxy params like below:

params:
  proxy:
    address: YOUR-PROXY-IP:YOUR-PROXY-PORT

3. It is recommended that you update the source attribute to a reasonable value, for example, to your cluster name.

params:
  ...
  source: my-cluster

4(Optional). If Istio is deployed in non default namespace replace istio-system with namespace name into which Istio is deployed.

Example: Change namespace: istio-system to namespace: istio-demo, handler: wavefront-handler.istio-system to handler: wavefront-handler.istio-demo

See the reference docs for the available configuration parameters.

Deployment

Installation

Execute the following command to configure the Istio Mixer to publish metrics to Wavefront using this adapter. This step must be performed after deploying Istio.

$ kubectl apply -f config.yaml

You should now be able to see Istio metrics on Wavefront under your configured source (or istio by default).

Uninstallation

To uninstall this adapter, use the following command.

$ kubectl delete -f config.yaml

Contributing

Please see CONTRIBUTING.md if you'd like to contribute.

Troubleshooting

• Check Istio adapter logs for errors kubectl logs wavefront-xxxxxxx-xxxx -n wavefront-istio.
• Check if Mixer is running kubectl -n istio-system get service istio-telemetry. If the pod istio-telemetry is not running then enable the Mixer.
• If Wavefront proxy is configured with the adapter then check proxy logs for errors kubectl logs wavefront-adapter-for-istio-proxy-xxxxxxx-xxxx -n wavefront-istio.

License

Wavefront by VMware Adapter for Istio is licensed under the Apache License, Version 2.0. See LICENSE for the full license text. Also, see the open_source_licenses file for the full license text from the packages used in this project.

wavefront-adapter-for-istio's Issues

Add Integration Tests

Describe the feature request
Currently, there are only a few unit tests present in the project. It'd be good to add end-to-end tests that run as a part of CI checks.

Additional context
Wavefront has a queryApi defined which could be leveraged to achieve this. Also, see #30.

Formatting is off for the adapter page on Istio.io

The formatting for the VMWare Wavefront adapter is odd.

https://preliminary.istio.io/docs/reference/config/policy-and-telemetry/adapters/wavefront

The logo for adapter appears like a watermark on the table and is cut off. Other adapters don't seem to include logos, so maybe it's something special with that.

Multi-cluster bug

Describe the bug
Metric doesn't have any tag to indicate from which Kubernates/Istio cluster came, so, a customer have the same application running on the two or more clusters metrics are mix and there is no way to know from which cluster came.

Expected behavior
Metrics should have a tag, or source label, indicate from which cluster came.

Steps to reproduce the bug
Create 2 Istio cluster, deploy the same app on both, and deploy the adapter pointing to the same WF and with the same KEY.

Update helm to update API's used for Deployments and Sevices to apps/v1

Describe the bug
helm chart to installation will fail on kubernetes version 1.16 and above

With 1.16 version of kubernetes, Deployment in the extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions is no longer served.

In the new api i.e. apps/v1, for Deployment - spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades
Refer to https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

We Need to modify out helm charts to include spec.selector in Deployment, specifically modify it to something like :

---
# Source: wavefront/templates/adapter.yaml
# deployment for adapter wavefront
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wavefront
  namespace: wavefront-istio
  labels:
    app: wavefront
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wavefront
  template:
    metadata:
      labels:
        app: wavefront
      annotations:
        sidecar.istio.io/inject: "false"
        scheduler.alpha.kubernetes.io/critical-pod: ""
    spec:
      containers:
      - name: wavefront
        image: vmware/wavefront-adapter-for-istio:0.1.2
        imagePullPolicy: Always
        ports:
        - containerPort: 8000
--- 

Steps to reproduce the bug
{{ Minimal steps to reproduce the behavior }}
Just use existing helm chart to install the adapter on K8s version 1.16 and above.

Version
{{ What version of Istio and Kubernetes are you using? Use istioctl version and kubectl version }}
This can be seen using any kubernetes version 1.16 and above.

Distribute Standalone Adapter Binary

Describe the feature request
Currently, the adapter is distributed via a Docker image and is installed by invoking Helm or Kubernetes commands. Istio allows users to integrate services that run on virtual machines into the service mesh. Therefore, it'd be useful to publish the adapter binary as a standalone application to be deployed outside of a Kubernetes cluster i.e. directly on a host OS.

Describe alternatives you've considered
Kubernetes and Helm deployment configurations.

Add support for Istio 1.1.x

Describe the feature request
We should add support for integrating this adapter with Istio 1.1.x (both 1.1.0 and 1.1.1). We can do this incrementally in 2 separate PRs, one following the other, and publish 2 releases in the process.

Additional context
Go Modules - https://github.com/golang/go/wiki/Modules
Istio 1.1.x releases - https://github.com/istio/istio/releases
Adapter guide - https://github.com/istio/istio/wiki/Mixer-Out-Of-Process-Adapter-Walkthrough

No TCP metrics

Describe the bug
There is no metrics about the TCP (Network) metrics.

Namespace on the metrics name

Describe the bug
The Istio namespace (usually istio-system) is part of the metrics name, but that name is defined by the user, so if a user change it will we get different metrics and the dashboards will not work.

For example, I did configure a Istio instance with the default namespace, and another one with glaullon-system as namespace, and I get these two metrics:

• istio.requestcount.instance.istio-system.count
• istio.requestcount.instance.glaullon-system.count

Only the first one will appear on the dashboard.

Of course I had to modify lots of adapter files to make it work with the new namespace , but I didn't had to modify any of the Istio Booking demo files to deploy it. So the adapter files should not depends on the namespace, if that is possible

Expected behavior
The namespace should be a tag of the metrics and not part of the name, and the adaptar should take the namespace from the environment.

How to send application metrics to Wavefront using ISTIO

Hi,
My usecase is I have a spring boot project running in k8s with ISTIO setup. I want to send spring boot application metrics published at /metrics endpoint to wavefront using Istio. Prometheus provides a way to do this with some annotations as mentioned here. Does wavefront with istio also provide a similar feature ? Please direct me how I can achieve sending my application metrics to wavefront.

Create A Stable Installation Script

Describe the feature request
Currently, there are 2 ways of installing the adapter i.e. by curling the manifest YAML or the released Helm chart. Also, installation procedure varies by patch versions i.e. 0.1.0 and 0.1.1. Also, since the master branch is used for active development, we currently rely on the 0.1.0 tag for stable installation instructions. All of this creates room for confusion.

Therefore, we should have a dowloader script at an accessible location i.e. on the master branch which can automatically download configuration from the latest stable release.

Additional context
See the Istio installation instructions for an example.

Create GitHub Pages

Describe the feature request
Currently, the documentation is served only via README files. It'd be good to have GitHub Pages (via the gh-pages branch) for this project to keep the documentation structured.

Describe alternatives you've considered
Put everything in READMEs.

Additional context
GitHub Pages - https://pages.github.com/

Helm Chart Credentials values confusion

Describe the bug
When configuring a helm deployment to use proxy access to wavefront via a values file created outside of the chart (using -f flag) and only the proxy credentials are specified in specified values file:

credentials:
  proxy:
    address: PROXY-HOST:PROXY-PORT

The proxy configuration is not applied to the resulting deployment. Instead, the direct credentials from the default values.yaml defined in the chart is used.

Currently, in order to configure proxy communication, the cred samples in the default values.yaml in the root of the chart have to be removed:

Expected behavior
If only proxy credentials are specified in a values file specified on the helm command, proxy communication to wavefront should be configured

Steps to reproduce the bug
Specify proxy credentials in a values.yaml file outside of the chart and use it with a helm deploy command.

Version
Istio 1.0.6, Kubernetes 1.11.6

Installing the adapter via helm fails with "Error: no available release name found"

Downloaded and installed istio. Installing the adapter returns the below error:
$ helm install install/wavefront/
Error: no available release name found

Expect the above to install the adapter successfully.

kubectl version:
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.6", GitCommit:"9b635efce81582e1da13b35a7aa539c0ccb32987", GitTreeState:"clean", BuildDate:"2018-08-16T21:33:47Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

istioctl version:
Version: 1.0.0
GitRevision: 3a136c90ec5e308f236e0d7ebb5c4c5e405217f4
User: root@71a9470ea93c
Hub: gcr.io/istio-release
GolangVersion: go1.10.1
BuildStatus: Clean

Provide a mechanism to safely store Wavefront credentials

Describe the feature request
Currently, the Wavefront credentials are stored transparently in the YAML files. This can pose a security risk, and it'd be good to provide a secure alternative for the same.

Describe alternatives you've considered
Store the credentials in a separate volume to be provided at runtime.

Additional context
See istio/istio#4434.

Migrate to go-metrics-wavefront Histograms

Describe the feature request
Currently, we use rcrowley/go-metrics to create Histograms and use go-metrics-wavefront to send those over to Wavefront. Once go-metrics-wavefront exposes a new Histogram implementation (based on the wavefront-sdk-go package), we should update the implementation to leverage that instead.

Documentation: Document the various config parameters and their use case

Need to document the configuration params in https://github.com/vmware/wavefront-adapter-for-istio/blob/master/install/wavefront/values.yaml#L15 in more detail. (especially for "prefix")

Support deploying into alternate namespaces

Describe the bug
Istio doesn't require that you install it to istio-system. It can be deployed under another namespace. However the connector/helm is currently hardcoded to istio-system.

Expected behavior
The helm would expose a template variable allowing the user to override the namespace, and have it apply uniformly to the adapter deployment.

Vendor using Go Modules

Describe the feature request
Golang introduced preliminary support for Modules in 1.11 which is set to be enhanced in the upcoming v1.12 release. We should move away from the third-party vendoring tool to the language provided vendoring mechanism.

Describe alternatives you've considered
There have been a few issues with the current vendoring tool, Glide, for example, see this comment on PR #51.

Setup CI

Describe the feature request
The PR workflow doesn't include any sort of testing, and it'd be good to have CI checks in place.

No 'requestduration' metrics

Describe the bug
On the sample_operator_config.yaml file there is 4 metrics defined:

    - requestsize
    - requestcount
    - requestduration
    - responsesize

But in WF I don't see any requestduration metrics.

Only expose a single namespace variable in Helm configuration

Describe the bug
As described in this comment, the Helm configuration currently accepts 2 namespace parameters as shown below.

https://github.com/vmware/wavefront-adapter-for-istio/blob/646ea4e89e711eea58e1ef0c1c0d7784deffa51f/install/wavefront/values.yaml#L15-L17

Currently, there's a bug in the adapter/Istio that causes the adapter to report no metrics when the entire Helm chart/yaml is deployed onto a single namespace other than istio-system.

Expected behavior
We should accept only a single namespace parameter (instead of namespaces.istio and namespaces.adapter) and deploy everything from the Helm chart/yaml onto that namespace.

Steps to reproduce the bug
Use the YAML configuration from the 0.1.0 or 0.1.1 versions of the adapter and update the namespace to something other than istio-system. You should see no metrics and logs flowing through the adapter.

Add support for Istio 1.3.x

Describe the feature request
1.3.x has been GA for a while and the wavefront adapter adapter doesn't officially support that yet.
Please provide an installation yaml file that will work with istio 1.3.x

Describe alternatives you've considered
n/a

Histograms need to be clear after x seconds

Describe the bug
On histograms metrics we are sending the values (mean, max, min, etc...) calculated over the life time of the adapter, and that it's causing for example to show an average request duration when really there is no requests and it should be 0 or no data.

Expected behavior
Send the histogram calculated values for the last X seconds, in other words, clear the histogram data every X seconds. I think it should be between 30 or 60 seconds by default, but could be also useful to give user the posibilito to change it.

Add a Helm Chart

Describe the feature request
Currently, this adapter is installed by manually downloading and editing configuration YAMLs. It'd be better to support Helm based installations too.

Wrong values on `requestcount` metrics

Describe the bug
The values of the requestcount metrics are showing incorrect values, looks like is counting each request twice, If I made 1000 request, the counter shows 2000

Version
istio-1.1.0.snapshot.1