krzko / run-o11y-run

A single-binary 🌯 wrapper around `docker compose` with embedded configurations to effortlessly run your local observability stack

run-o11y-run

_____ _____ _____     _____ ___   ___   __ __     _____ _____ _____
| __  |  |  |   | |___|     |_  | |_  | |  |  |___| __  |  |  |   | |
|    -|  |  | | | |___|  |  |_| |_ _| |_|_   _|___|    -|  |  | | | |
|__|__|_____|_|___|   |_____|_____|_____| |_|     |__|__|_____|_|___|

A single-binary 🌯 wrapper around docker compose with embedded configurations to effortlessly run your local observability stack.

The underlying observability stack is built on Grafana products and OpenTelemetry. It includes the following services:

• Grafana
• Grafana Loki
• Grafana Tempo
• MinIO
• OpenTelemetry Collector
• Prometheus
• Pyroscope

Prerequisites

run-o11y-run depends on the latest version of Docker Desktop, which includes the docker compose command.

Quick Start

Install

brew

Install brew and then run:

brew install krzko/tap/run-o11y-run

Download Binary

Download the latest version from the Releases page.

Run

• Run run-o11y-run
• Configure your service to push telemetry data to one of the following endpoints. This can be done by setting the OTEL_EXPORTER_OTLP_ENDPOINT environment variable or updating your config file. Make sure to set the traffic to insecure:
  • OTLP (grpc): http://localhost:4317
  • OTLP (http): http://localhost:4318
  • Jaeger: http://localhost:14268
  • Zipkin: http://localhost:9411
• Logs are processed via two means:
  • Tailed from /var/log/*.log and ./*.log on your local machine.
  • A Syslog RFC 3164 header format, syslog receiver operates on localhost:8094
• Profiling data can be pushed to http://localhost:4040
• To exit gracefully, press CTRL+C.

Commands

run-o11y-run is a powerful command-line tool that provides seamless management of your local observability stack. It offers three simple commands: start, stop, and clean.

$ run-o11y-run start

✨ Starting...
[+] Running 56/39
 ✔ tempo 5 layers [⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                           142.9s
 ⠦ minio 9 layers [⣿⣿⣿⣿⣿⣄⣿⣿⣿] 48.23MB/96.92MB Pulling                                                                                                      170.6s
 ⠦ otel-collector 6 layers [⣿⣿⣿⣿⣿⣷] 45.09MB/48.15MB Pulling                                                                                                170.6s
 ⠦ grafana 12 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulling                                                                                                170.6s
 ⠦ prometheus 15 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulling                                                                                          170.6s
 ✔ loki 9 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                         81.8s

start Command

The start command allows you to launch run-o11y-run containers and start your observability stack effortlessly. You can customise the behaviour using various flags.

One such flag is the --registry flag, which enables you to specify a Docker Registry from which to pull the required images. By default, run-o11y-run uses Docker Hub as the registry. Here's an example of using the --registry flag:

run-o11y-run start --registry <registry-url>

Replace <registry-url> with the URL of your desired Docker Registry.

To further enhance your setup, you can also utilise the --external-network flag, which enables integration of your own docker-compose configurations with run-o11y-run. This allows you to combine the services of run-o11y-run with your existing infrastructure seamlessly.

To start run-o11y-run in detached mode, use the --detach flag. This will start the containers in the background.

The --yolo flag can be used with the run-o11y-run command to apply the :latest tag to all images in your stack. This flag allows you to conveniently run the latest versions of the images without specifying the specific tags.

• NOTE: However, please note that using the --yolo flag may introduce potential risks, as it may lead to compatibility issues or unexpected behaviour if the latest images have breaking changes or dependencies

For more details on using the --external-network flag, refer to the External Network guide.

patch Command

The patch command in run-o11y-run provides a convenient way to modify your own docker-compose.yaml file, allowing you to establish a network bridge between the o11y stack and your customer stack. This feature is particularly useful when you have started the o11y stack with the --external-network flag, enabling seamless integration with your existing infrastructure.

See the extended patch documentation for more detailed information.

stop Command

The stop command is used to gracefully stop the run-o11y-run containers. It ensures a clean shutdown of your observability stack. Here's an example of using the stop command:

run-o11y-run stop

open Command

The open command allows you to conveniently open various services provided by run-o11y-run in your default web browser. This feature saves you time by quickly launching the relevant service pages.:

run-o11y-run open --service <loki|tempo|prometheus|prometheus-direct|pyroscope-direct>

Ensure that run-o11y-run is already running before using the open command.

Note: Make sure your default web browser is properly configured on your system for this command to work.

clean Command

The clean command is used to stop and remove run-o11y-run containers, files, and networks. It helps you clean up your environment after using run-o11y-run. Here's an example of using the clean command:

run-o11y-run clean

ports Command

The ports command allows you to list the available ports used by the application. It provides a convenient way to check which ports are used by various services within the observability stack.

Here's an example output of the ports command:

+-----------+-------------------+
|   PORT    |      SERVICE      |
+-----------+-------------------+
| 3000/tcp  | Grafana           |
| 3100/tcp  | Loki              |
| 4040/tcp  | Pyropscope        |
| 4317/tcp  | OTLP (gRPC)       |
| 4318/tcp  | OTLP (HTTP)       |
| 8094/tcp  | Syslog (RFC3164)  |
| 9090/tcp  | Prometheus Direct |
| 9411/tcp  | Zipkin            |
| 14268/tcp | Jaeger            |
+-----------+-------------------+

Local Service Links

• Grafana Loki
• Grafana Tempo
• Grafana Prometheus
• Prometheus Direct
• Pyroscope Direct

Documentation

• FlameQL
• LogQL
• PromQL
• TraceQL

Troubleshooting

run-o11y-run is built on top of Docker, and if you encounter any issues or things don't seem to be working as expected, please use the standard Docker debugging techniques.

Make sure you run run-o11y-run -clean to clean up the configurations before attempting any troubleshooting steps.

In case you need further assistance, refer to the Docker documentation and Docker troubleshooting guide.

run-o11y-run's Issues

Integration with KIND (Kubernetes in Docker)

Description

I would like to request the integration of run-o11y-run with Kind (Kubernetes in Docker). This feature would allow users to easily deploy their observability stack using Kubernetes manifests generated from the docker-compose.yml file used by run-o11y-run.

Currently, run-o11y-run provides a convenient way to run a local observability stack using Docker Compose. However, as many developers and teams are adopting Kubernetes for local development and testing, having the ability to deploy the stack as Kubernetes resources would be highly beneficial.

Proposed Solution

The proposed solution is to enhance run-o11y-run with a new command or flag that generates Kubernetes manifests from the existing docker-compose.yml file. These generated manifests can then be used to deploy the observability stack using Kind or any other Kubernetes cluster.

The integration could include the following steps:

• Parsing the docker-compose.yml file and extracting the relevant services, volumes, and network configurations.
• Converting the extracted configurations into Kubernetes manifest files (e.g., Deployment, Service, ConfigMap).
• Providing an option to output the generated manifests to a specified directory.
• Optionally, supporting the deployment of the generated manifests to a Kind cluster directly from run-o11y-run.

Expected Behaviour

After integrating run-o11y-run with Kind, users would be able to generate Kubernetes manifests from their existing docker-compose.yml file and deploy the observability stack to a local Kind cluster. This would provide a seamless transition for developers and teams who are already using Kind for local Kubernetes development.

Additional Context

Kind (Kubernetes in Docker) has gained popularity as a lightweight and fast option for running local Kubernetes clusters. Integrating run-o11y-run with Kind would align with industry trends and make it easier for users to adopt and leverage Kubernetes for their observability needs.

refactor code to replace `map[any]any` with actual struct which represents docker-compose manifest

dealing with map[any]any and casting particular fields to particular types doesn't seem to be the best solution.
It would be nice to use some structs to manage docker-compose manifests in more organised way.

Ability to Run Specific Services

Description

I would like to request the addition of a feature in run-o11y-run that allows users to selectively run specific services from the observability stack. Currently, run-o11y-run starts all the services defined in the docker-compose.yml file. However, there are scenarios where running only a subset of services would be more efficient or necessary.

Proposed Solution

The proposed solution is to enhance run-o11y-run with the ability to specify which services to run when executing the command. This can be achieved by introducing a new flag or argument that accepts a comma-separated list of service names. When this flag is provided, run-o11y-run should only start the specified services and their dependencies, ignoring the rest.

Expected Behaviour

With this enhancement, users can have finer control over the execution of the observability stack. For example, running only the essential services during development or testing phases, or isolating specific services for troubleshooting or performance analysis purposes.

Additional Considerations

• It would be beneficial to provide an option to list the available services in the docker-compose.yml file, making it easier for users to identify the service names they want to run.
• Proper error handling and informative messages should be implemented in case of invalid service names or missing dependencies.

This feature would enhance the flexibility and usability of run-o11y-run, allowing users to tailor their observability stack to their specific needs.

Extract as a Framework for Running Custom Stacks

Description

I would like to propose the extraction of the run-o11y-run tool into a reusable framework that can be used to run custom stacks. Currently, run-o11y-run is specifically tailored for running the provided observability stack, but it would be beneficial to make it more versatile and adaptable to different stack configurations.

Proposed Solution

The proposed solution is to refactor run-o11y-run into a framework that can be easily extended and configured to run custom stacks. The framework should provide the necessary abstractions, APIs, and hooks to allow users to define their own stack configurations, including services, dependencies, and environment variables.

Key Features of the Framework:

• Modularity: The framework should be designed to support pluggable components, allowing users to easily add or remove services based on their requirements.
• Configuration: The framework should provide a flexible configuration mechanism that enables users to define their stack configuration in a declarative manner, using compose.yaml or docker-compose.yaml file.
• Extensibility: Users should be able to extend the framework with custom logic or components to integrate additional services or modify the behavior of existing services.
• Lifecycle Management: The framework should handle the startup, shutdown, and monitoring of services, ensuring proper handling of dependencies and clean termination.

Expected Benefits

By extracting run-o11y-run into a reusable framework, users will have the freedom to define and run their own stacks tailored to their specific needs. This framework can serve as a foundation for creating custom stack configurations, integrating various monitoring and observability tools seamlessly.

Additionally, the availability of the framework will enable the community to contribute and share their custom stack configurations, fostering collaboration and innovation within the observability space.

Implementation Considerations

• The framework should provide clear documentation and examples to guide users in defining their custom stack configurations.
• Compatibility with existing Docker Compose files or Kubernetes manifests could be considered to facilitate the migration of existing stack definitions.