couchbase / cbmonitor Goto Github PK

I eventually figured this out when ns_collector gave me the message:

seriesly.exceptions.ConnectionError: Connection refused to "http://127.0.0.1:3133/"

But it would be nice if the README made this more clear up front.

Traceback (most recent call last):
File "cbtop", line 9, in
from libcbtop.main import main
File "priority15/libcbtop/main.py", line 9, in
import visit_cb as vc
File "priority15/libcbtop/visit_cb.py", line 11, in
from tabula.table import Table
File "virtualenvs/priority15/lib/python2.6/site-packages/tabula/table.py", line 6, in
from section import Section
File "virtualenvs/priority15/lib/python2.6/site-packages/tabula/section.py", line 323
{k:v-1 for k,v in self.irt.iteritems() if v > row_num})
^
SyntaxError: invalid syntax

Currently this is defined manually:

function (doc, meta) {
    emit(meta.id, doc);
}

And it must be integrated into dashboard functionality.

I've got a workflow that seems to work for me, but it involves a lot of deleting everything and starting over, and I don't know if thats right. Typically right now I do:

1. Delete old seriesly db
2. Start seriesly
3. Delete cbmonitor.db
4. Run bin/webapp syncdb
5. Run bin/webapp runserver
6. Run bin/ns_collector laptop.cfg
7. Do interesting things and look at data

To start new test, start over at number 1.

If I deviate from these steps I usually get data from previous tests showing up, or errors about cluster names already existing, collectors crashing, etc.

Right now cluster tree looks like

cluster
      |_
        server
              |_
                 bucket

which is wrong because bucket is part of cluster.

I created my own config file and ran:

bin/ns_collector laptop.cfg

This appears to do something, a cbmonitor.couch file shows up, and the file size grows over time.

Next I started web UI:

bin/webapp runserver

When I view http://localhost:8000/ there are no clusters visible. In the demo you said something about normally not needing to add them manually. Is there another step I should perform here instead?

Next I went ahead and tried adding it manually. I added the information about the cluster running on my laptop. It seems to discover all the correct information about this cluster. Next if I click over to the Metrics area, select a cluster, node, and the first bucket "beer-sample". I see only one metric:

swap usage

If try to drag it to the view window on the right, nothing happens.

If select any other bucket, I don't see any metrics at all.

1. Is this the correct way to link the web ui to the statistics?
2. Does this point to a problem in the collection process or the way I'm using the web frontend?

I see query for one of our metrics going to seriesly as:

http://localhost:3133/cbmonitor/_query?group=1000&ptr=/metric/laptop/beer-sample/127.0.0.1/curr_items_tot&reducer=avg

In my seriesly db this returns data like:

{
"1361547184000": [
null
],
"1361547194000": [
null
],
"1361547204000": [
null
],
"1361547214000": [
null
],
"1361547225000": [
null
],
"1361547227000": [
7303
],
"1361547235000": [
null
],
"1361547237000": [
7303
],
"1361547245000": [
null
],
"1361547248000": [
7303
],
"1361547255000": [
null
],
"1361547259000": [
7303
],
"1361547265000": [
null
],
"1361547269000": [
7303
],
"1361547275000": [
null
],
"1361547280000": [
7303
],
"1361547285000": [
null
],
"1361547291000": [
7303
],
"1361547295000": [
null
],
"1361547301000": [
7303
],
"1361547305000": [
null
],
"1361547312000": [
7303
],
"1361547315000": [
null
],
"1361547323000": [
7303
],
"1361547326000": [
null
],
"1361547335000": [
7303
],
"1361547346000": [
7303
]
}

I believe the large number of nulls is because I have many different kinds of metrics being stored (also from multiple clusters).

What if we stored data more like:

{
"meta": {
"cluster": cluster-name (optional)
"node": node-name (optional)
"bucket": bucket-name (optional)
},
"metric": {
...
}
}

Then in we'd use both ptr and filters in the query:

ptr=/metric/curr_items_tot
f=/meta/cluster
fv=laptop
f=/meta/bucket
vb=beer-sample
f=/meta/node
fv=127.0.0.1

This would also be more extendable for different kinds of metric meta-data in the future.

Also would like to ping @dustin for input on this.

Baseline builds, sorting preference and number of items per page.

If I run this command to list stats keys and filter:

curl http://localhost:8091/pools/default/buckets/beer-sample/stats | jq '.op.samples | keys' | grep replication

I see the following:

"replication_active_vbreps",
"replication_changes_left",
"replication_commit_time",
"replication_data_replicated",
"replication_docs_checked",
"replication_docs_rep_queue",
"replication_docs_written",
"replication_num_checkpoints",
"replication_num_failedckpts",
"replication_size_rep_queue",
"replication_waiting_vbreps",
"replication_work_time",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/active_vbreps",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/changes_left",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/data_replicated",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/docs_checked",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/docs_rep_queue",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/docs_written",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/num_checkpoints",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/num_failedckpts",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/size_rep_queue",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/time_committing",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/time_working",
"replications/f4268e62702130298bf87f17cc481219/beer-sample/bs/waiting_vbreps",

However, none of these show up in the UI.

Currently we plot all charts, this is too expensive. It takes time to create even empty PNG image. We must be more selective.

For instance, we can exclude stats for all nodes by default and plot only cluster-wide metrics. Also we need "report presets" with a list of typical metrics that must be plotted. Like "View performance test", "XDCR performance test", "System test", and etc.

refactoring

http://review.couchbase.org/#/c/24795/

Current rest api supports one metric per request:

build=2.0.0-1723-rel-enterprise&metric=Latency, ms&value=555

I'm running Chrome 24.0.1312.57, when I drag metric to right-hand side nothing happens.

When I switched to Firefox it works correctly.

It seems that the order I add items to a chart affects the display, is this expected?

In the 4 panel chart attached:

top-left and top-right are the two values plotted on their own
bottom-left and bottom-right show the different charts depending on which one is added first

The specific case I have in mind when we create an new XDCR, new stats are introduced for this. I think with the current model there is no way to discover these new metrics once data collection has started.

2013-02-21 10:45:26 - WARNING - {"name": "Ensure this value has at most 64 characters (it has 75)."}
[21/Feb/2013 10:45:26] "POST /cbmonitor/add_metric_or_event/ HTTP/1.1" 400 68

2013-02-22 07:33:41 - WARNING - columns name, cluster_id, server_id, bucket_id are not unique
[22/Feb/2013 07:33:41] "POST /cbmonitor/add_metric_or_event/ HTTP/1.1" 400 61