streamnative / apache-pulsar-grafana-dashboard Goto Github PK

Dashboard:

When I run gemerate_dashboards.sh I got this error message.

➜  apache-pulsar-grafana-dashboard git:(master) ./scripts/generate_dashboards.sh prometheus-test.example.com pulsar-cluster-test
Setting PULSAR_PROMETHEUS_URL from script input.
PULSAR_PROMETHEUS_URL set to: prometheus-test.zhenguanyu.com
Setting PULSAR_CLUSTER from script input.
PULSAR_CLUSTER set to: pulsar-cluster-test
Using existing PULSAR_CUSTOM_PROMETHEUS environment variable.
PULSAR_CUSTOM_PROMETHEUS set to:
Using existing GF_LOKI_URL environment variable.
GF_LOKI_URL set to:
Using existing GF_LOKI_DATASOURCE_NAME environment variable.
GF_LOKI_DATASOURCE_NAME set to:
Generating Datasources ...
./scripts/generate_dashboards.sh: line 105: j2: command not found

It should be nice to configure CORS allowed hosts from backend configuration in case of splitted frontend/backend runtimes and host.

Error log:

root@c8faefc852da:/apache-pulsar-grafana-dashboard# ./scripts/generate_dashboards.sh http://172.17.0.2:9090 standaloneGenerating Datasources ...
Traceback (most recent call last):
  File "/usr/local/bin/j2", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/j2cli/cli.py", line 206, in main
    sys.argv[1:]
  File "/usr/local/lib/python3.6/dist-packages/j2cli/cli.py", line 186, in render_command
    result = renderer.render(args.template, context)
  File "/usr/local/lib/python3.6/dist-packages/j2cli/cli.py", line 87, in render
    .render(context) \
  File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.6/dist-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/apache-pulsar-grafana-dashboard/target/datasources.yml", line 65, in top-level template code
    - name: "{{ GF_LOKI_DATASOURCE_NAME }}"
jinja2.exceptions.UndefinedError: 'GF_LOKI_DATASOURCE_NAME' is undefined

Currently, the charts have limited support for observing multiple independent Pulsar clusters, based on the cluster label. My suggestion is that the charts be improved to support multiple BookKeeper clusters and ZooKeeper clusters, and to do it in a standards-based way.

Specifically my ask is that job not be treated as a constant (bookie, etc.) but as representing a unique cluster. For example, imagine that I have two separate BK clusters. There'd be two Prometheus jobs configured, one for each BK cluster. The name of the job would be user-defined and likely represents the name of the BK cluster (e.g. bookkeeperset/cluster-1). For simple setups, the job name would likely be bookie as before.

This approach seems to be standards-based. From Prometheus docs (reference):

In Prometheus terms, an endpoint you can scrape is called an instance, usually corresponding to a single process. A collection of instances with the same purpose, a process replicated for scalability or reliability for example, is called a job.

I observe that the Prometheus UI and the Prometheus operator seem to assume this sort of setup. For example, the UI paints a given job in red if any of the instances of that job are down. This would make no sense if the job consisted of instances across numerous independent clusters. Meanwhile the Prometheus operator generates job names based on the ServiceMonitor name (which would be service-specific).

In some cases it may be necessary to provide a 'job selector' widget on the dashboard, which would set the job variable. Most of the queries seem to use $job already as a filter, which is good.

./scripts/generate_dashboards.sh http://localhost:9090 pulsar-19990
Setting PULSAR_PROMETHEUS_URL from script input.
PULSAR_PROMETHEUS_URL set to: http://localhost:9090
Setting PULSAR_CLUSTER from script input.
PULSAR_CLUSTER set to: pulsar-19990
Using existing PULSAR_CUSTOM_PROMETHEUS environment variable.
PULSAR_CUSTOM_PROMETHEUS set to:
Using existing GF_LOKI_URL environment variable.
GF_LOKI_URL set to:
Using existing GF_LOKI_DATASOURCE_NAME environment variable.
GF_LOKI_DATASOURCE_NAME set to:
Generating Datasources ...
Traceback (most recent call last):
File "/usr/local/bin/j2", line 10, in
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/j2cli/cli.py", line 206, in main
sys.argv[1:]
File "/usr/local/lib/python2.7/dist-packages/j2cli/cli.py", line 186, in render_command
result = renderer.render(args.template, context)
File "/usr/local/lib/python2.7/dist-packages/j2cli/cli.py", line 87, in render
.render(context)
File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/mnt/h/apache-pulsar-grafana-dashboard/target/datasources.yml", line 68, in top-level template code
url: "{{ GF_LOKI_URL }}"
jinja2.exceptions.UndefinedError: 'GF_LOKI_DATASOURCE_NAME' is undefined

Need to check or improvement the doc to start a new dashboard. currently, some steps may have issue, and made user not able to add it.

I am browsing the variables and I can see that POD is listed in the query

ifconfig | grep "inet " | grep -v 127.0.0.1 | awk '{ print $2 }

I 'm running this command in Ubuntu 16.04 , the result is

addr:172.17.0.1
addr:192.168.247.129.

It doesn't just contain ip.

Angular plugins will be removed in future release, it's easy to switch panels to not-deprecated versions:
https://grafana.com/docs/grafana/latest/developers/angular_deprecation/angular-plugins/#automatic-migration-of-plugins

I have already using the grafana dashboard, now i need to import the Pulsar template. Is there any Template ID is Available or How to Use this Template in the Existing Grafana Dashboard ?

If datasource field of variables or panels is empty, the dashboard will be invalid.
Seems jinja2 with following version doesn't render "{{ PULSAR_CLUSTER }}" correctly.

➜ jinja2 --version
jinja2-cli v0.8.2
 - Jinja2 v3.1.2

In the panel pulsar-topics, the query for cluster/namespace/topic may hang prometheus.

It will query all the metrics to find all the cluster/namespace/topics, which may cause prometheus crash.

Hi,

how to configure when we have multiple pulsar cluster running? What should be there for the external_labels:cluster settings?

Thanks for advice,

Tom

I have run docker image of streamnative/apache-pulsar-grafana-dashboard and I just got some error Datasource named ${DS_default} was not found when I access the topic dashboard, have something missed?
Thanks any help!

i refer to officail Node Metric dashboard and according iostat output
found metric I/O Util, I/O Average Write Time, I/O Average Read Time is not right as folllowing

I/O Util

vs

I/O Average Read Time

vs

We have removed the AspectJ based metrics for ZooKeeper, we need to update the Zookeeper dashboard for Pulsar version which is greater than 2.8

why have a problem :datasource named {{ PULSAR CLUSTER}} was not found ,when import dashboard

1. current support version 7.3 is too old
2. remove dependencies of loki.