livingdocsio / monitoring

Grafana, Prometheus, Loki, Jaeger and Opentelemetry in an easy-to-use docker compose example setup to extract logs, metrics and traces in development.

A simple Loki setup with Grafana

This setup uses Loki for logs, Prometheus for metrics, and Jaeger for tracing. It will automatically collect logs from other Docker containers running on the same server if the installation steps below are followed.

The Livingdocs Server can be configured to send metrics data to the stack, and there is also a way to send log data when running the service without the use of Docker (for local development purposes). More details can be found in the Livingdocs Server telemetry documentation.

The stack includes the following:

• cAdvisor [Docs] [Local UI] - Provides resource usage and performance metrics of Docker containers to Prometheus
• Grafana [Docs] [Local UI] - UI to explore logs and metrics using queries, charts, and alerts
• Jaeger [Docs] [Local UI] - Provides tracing data which is linked to each incoming Livingdocs server request
• Loki [Docs] - Ingests logs which can be viewed and queried from within Grafana
• OpenTelemetry Collector [Docs] - Collects metrics data from Livingdocs server and exports the data to Prometheus
• Prometheus [Docs] [Local UI] - A monitoring toolkit for timeseries based metrics
• Vector [Docs] - Transforms Docker logs and send them to Loki, as well as collecting logs from local Node.js processes if required

Installation

• Clone the repository and launch cAdvisor, Grafana, Jaeger, Loki, OpenTelemetry Collector, Prometheus, and Vector:

  git clone [email protected]:livingdocsIO/monitoring.git
  cd monitoring
  docker-compose up -d

• Navigate to http://localhost:3000/ and log in using the default Grafana user (admin/admin).

How to add livingdocs-server logs to Grafana/Loki

Every log in /data/logs/*.log will be pushed to Grafana/Loki. You can start the Livingdocs Server and pipe the logs to the logs folder.

Example: logs__pretty=false npm run start | tee ~/livingdocs/monitoring/data/logs/livingdocs-server.log

Then you should see the logs with a Loki Query: {file="/logs/livingdocs-server.log"}

Troubleshooting

Directory permissions

If you see any errors about being unable to create files or directories when starting the containers then the issue is probably related to user permissions. We have chosen to run the containers which write to disk with uid 1000 and gid 1000. We see this as being a sensible default for most users, but if it causes problems you can either change the user: "1000:1000" lines in docker-compose.yaml to values which suit your environment, or remove the lines and set the default container permissions on the directories:

chown -R 472:472 ./data/grafana
chown -R 0:0 ./data/jaeger
chown -R 10001:10001 ./data/loki
chown -R 65534:65534 ./data/prometheus

Jaeger EMSGSIZE error

Jaeger needs higher UDP package sizes than the maximum configured on Mac. The default value should be increased using:

sudo sysctl net.inet.udp.maxdgram=65536

monitoring's Issues

Hi, where to get the golang demo?

Hi, I want to study how to send metrics and logs and traces in golang project.
But I does not find the livingdocs project.
I want to know how to get a demo source, thank you very much.

Nit: retention_period should be a multiple of db 'period'

From Loki docs:

# How far back tables will be kept before they are deleted. 0s disables
# deletion. The retention period must be a multiple of the index / chunks
# table "period" (see period_config).
[retention_period: <duration> | default = 0s]

So something like:

  retention_period: 336h

This is just a drive-by comment, feel free to ignore, but thought it might save you some debugging in the future.

Cheers! :)