liatrio / o11y-demo

Hosts the files to demo o11y

o11y-demo

This is a self contained demo of the Liatrio OpenTelemetry Collector and Grafana using Prometheus as the data source. The intent of this demo is to show how to use the Liatrio OpenTelemetry Collector to scrape data from GitHub and expose it to Grafana for visualization. This demo is not intended to be used in production.

There is also a bonus section that shows a local development workflow for making changes to the Liatrio OpenTelemetry Collector utilizing Grafana to visualize and Promethues as the data store and a locally built collector image.

• o11y-demo
  • Architecture
  • Pre-requisites
  • Getting Started
  • Bonus Section: Development Workflow
  • Bonus Section: DORA Metrics with GitHub Event Logs
    • Requirements
  • TODO

Architecture

stateDiagram-v2
    Github --> Collector: Pushes
    Collector --> Github: Scrapes
    Collector --> Prometheus: Pushes
    Collector --> Loki: Pushes
    Grafana --> Prometheus: Pulls
    Grafana --> Loki: Pulls
    

Loading

Pre-requisites

• The Liatrio OpenTelemetry Collector images are hosted on ghcr.io see here for instructions on how to login
• The runtime Collector requires a GitHub Personal Access Token with the following configurations (Can be fine-grained or classic):
  • repo:All
  • read:packages
  • read:org
  • read:user
  • read:enterprise
  • read:project

?> This is a classic token configuration, fine grained might be easier

Getting Started

1. Clone this repo and navigate to the root directory
2. Create a .collector.env file in the root of the repo, make a copy of the example
3. Make sure organization is set to liatrio, unless you want to specify another
4. Run docker compose up
5. Open grafana by navigating to http://localhost:3000
6. To view the demo dashboard in Grafana go to dashboards > o11y > demo

?> You will need to wait a moment for Grafana to recieve data from Prometheus, the page will look like this until enough data has been recieved:

Once the data has been recieved it should look like this:

Service	Link
Grafana	http://localhost:3000
Prometheus	http://localhost:9090
Collector gRPC Receiver	http://localhost:4317
Collector HTTP Receiver	http://localhost:4318
Collector Prometheus Metrics	http://localhost:9464/metrics
Collector Health Check	http://localhost:8888/metrics
Webhook Event Receiver	http://localhost:8088/events

Bonus Section: Development Workflow

When you want to test pre-release functionality from the liatrio collector or test newly developed functionality you can use the following workflow to test your changes locally in a full stack environment.

1. Clone the liatrio-otel-collector repo
2. Run make dockerbuild in the root of the liatrio-otel-coollector repo
3. Follow Getting Started section but uncomment OTEL_COLLECTOR_IMAGE variable in the .env file
4. Run docker compose up in the root of this repo
5. For a more detailed developer workflow read the docs in the liatrio-otel-collector repo

Bonus Section: DORA Metrics with GitHub Event Logs

1. Create a GitHub App with permissions for Issues Deployments and Pull requests then have it subscribed to Issues Deployment Status and Pull request events while leaving webhooks disabled for now. This can be done by navigating to Settings->Developer Settings->GitHub Apps.

GitHub App Permissions	Github App Event Subscriptions

3. Using Ngrok or another tool to forward traffic from your GitHub App to your local machine, set up forwarding to http://localhost:8088/ which is going to be the endpoint for our webhook receiver should it be running locally.
4. This is going to give you a web address which we will be using as our webhook url in our GitHub App. Be sure to add /events to that address as that is the path that the webhook event receiver will be expecting these event logs at by default.
5. Uncomment the relevant code in the collector-config.yaml and docker-compose file in order to setup the tools required for this.
6. Now you should be all set to start ingesting GitHub event logs.

Requirements

1. Use GitHub Issues to track outages/interuptions in service caused by a recent deployment
2. There should only be one issue open at a time with a label called incident for all problems caused by the latest deployment
3. For GitHub to keep track of your deployments, you must be using GitHub Environments inside the workflow that runs the deployment
4. The workflow also has to be using tooling that deploys your code from GitHub to an external platform or GitHub itself such as Terraform

TODO

• consider publishing collector image with arch amd64 as latest tag
  • Decision was made to just use amd64 arch in the compose file for now
• add health check endpoint back in and update documentation
• dashboard template variables
• expand dashboard to include some more advanced expression to show off what you can do with granular data

o11y-demo's Issues

Prototype Deployment Frequency data collection

Create a means to collect the deployment events, and store them.

Required data:

• Deployments with timestamps

Tasks

Beta Give feedback

1. Pull the required data
2. Create a table for the log data

Create data pulling mechanism for DORA metrics

We will need to investigate how we are going to pull and store the data for the metrics.

We may be able to utilize some existing means, such as webhooks, or we may want to pull this data with our own GitHub app.

This should result in a means to pull the data, and store it for use in the dashboard.

Add explanation on o11y demo documentation for Token Permissions

Github Personal Access Token permissions need to be specified more accurately and avoid users from giving their token full permissions in case of potential issues.

Determine whether the metrics aspects should be contributed back to the upstream opentelemetry demo

Create DORA Dashboard

Create a dashboard to display DORA metrics in Grafana

Tasks

Beta Give feedback

1. Create Deployment Frequency table
2. Create Lead Time for Changes table
3. Create Change Failure Rate table
4. Create Time to Restore Service table

Prototype Change Failure Rate data collection

Collect data for Change Failure Rate, and store it

Required data:

• Deployments
• incident issues with timestamps

Tasks

Beta Give feedback

1. Collect data for metric
2. Create data relationships for metric

Document how to deploy new configurations to the OTEL collector in AKS/EKS

Add loki to the o11y demo & show the art of the possible for emitting logs

Overview

Having a log stack integrated into the o11y demo would provide good adoption value.

AC

• loki is added into the demo
• logs from the collector are emitted to loki & explored in grafana
• documentation is updated

Prototype Time to Restore Service data collection

Collect the data for the Time to Restore Service metric.

Required data:

• Issues tagged as incident with opening and closing timestamps
• Deployment with timestamp that closes the issue

Tasks

Beta Give feedback

1. Pull data for metric
2. Create table relationships for data

Prototype Lead Time for Changes data collection

Pull the data for Lead Time for Changes metric, and store it.

Required data:

• First commit for deployment time
• Deployment time

Tasks

Beta Give feedback

1. Pull the required data
2. Create tables for data relationships

Add traces to the o11y demo & show the art of the possible

Overview

Having a trace stack integrated into the o11y demo would provide good adoption value.

AC

• jaeger is added into the demo
• traces from the collector are emitted to jaeger is visualized into the demo
• documentation is updated

Investigate issue where ratio is being appended to metrics

In the demo, metrics are being appended with _ratio and are causing the dashboards not to correctly represent the data.