observiq-otel-collector-k8s

Configuration for instrumenting Kubernetes with the observIQ OpenTelemetry collector.

Support

Cluster Support

Most clusters should be supported. The following are tested:

On Prem / Minikube
Google Kubernetes Engine (GKE)
Amazon Elastic Kubernetes Service (EKS)
Azure Kubernetes Service (AKS)

Platform / Exporter Support

Most exporters should work. Some require additional configuration to work well (Google Cloud) while others support native OTLP (New Relic). The following are tested:

Google Cloud
New Relic
OTLP

Telemetry Types

Metrics, logs and traces are supported.

Metrics

K8s Metrics: Cluster, node, pod, and container metrics are collected.
App Metrics: Application specific metrics can be collected as well, see app/redis for an example.
Custom Metrics: If you have applications emitting metrics using an Open Telemetry SDK, they can be sent to the gateway collector at observiq-gateway:4317 (OTLP GRPC receiver).

Logs

Container logs are collected using the node agent.

If you have applications emitting logs using an Open Telemetry SDK, they can be sent to the gateway collector at observiq-gateway:4317 (OTLP GRPC receiver).

Traces

If you have applications emitting traces using an Open Telemetry SDK, they can be sent to the gateway collector at observiq-gateway:4317 (OTLP GRPC receiver).

Usage

⚠️ Please reference a tagged release, Do not rely on main branch to be reliable. If implementing this in a production environment, it is strongly recommended that you copy this code to your own deployment repository.

Prerequisites

OpenTelemetry Operator for handling collector deployment and configuration
Kustomize for deployment simplicity

Deploy Collectors

Installation

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

kustomize build environments/ci | kubectl apply -f -

kubectl apply -f app/redis/redis.yaml

New Relic

Deploy New Relic API Key (insert your api key into the command):

kubectl create secret generic newrelic-credentials \
    --from-literal=api-key=<api key here>

Update the override file in environments/newrelic/agent.yaml

Set K8S_CLUSTER environment variable to the name of your cluster.

Using Kustomize, deploy the New Relic configuration:

kustomize build environments/newrelic | kubectl apply -f -

OTLP

The OTLP configuration exposes the entire collector configuration to the user because OTLP is going to be unique for each environment.

Edit environments/otlp/agent_gateway.yaml and set the endpoint field (near the top) to the otlp endpoint you wish to send to. Make any additional changes required to match the OTLP destination you are sending to.
Using Kustomize, deploy the New Relic configuration:

kustomize build environments/otlp | kubectl apply -f -

agratae

Application Monitoring

Redis Example App

Redis will be our example application for metrics. When deployed, the OpenTelemetry operator will inject a collector container into each Redis pod, to collect metrics via localhost.

kubectl apply -f app/redis/redis.yaml

Sidecar injection is one method to collect application level metrics. Alternatively, the collector could be deployed as a single pod Deployment or Statefulset targeting the application's service. This would require the receiver to be "cluster aware". For example, Prometheus receiver with kubernetes detection (service discovery) or Elasticsearch (capable of collecting whole cluster metrics from a single endpoint).

micabe / observiq-otel-collector-k8s Goto Github PK