ciscodevnet / bigmuddy-network-telemetry-pipeline

bigmuddy-network-telemetry-pipeline

Archived

This project has not been supported in multiple years and has been pushed to an archive state. If anyone is interested in supporting it, contact [email protected] and we can arrange to un-archive it. The final version can be found on the "final" branch.

bigmuddy-network-telemetry-pipeline's Issues

Error: http2Server.HandleStreams failed to receive

Trying to get the NXOS telemetry to work with the pipeline, getting error:

2019/11/09 21:06:58 transport: http2Server.HandleStreams failed to receive the preface from client: EOF

on NXOS side:

  destination-group 2
    ip address 192.168.100.1 port 57500 protocol gRPC encoding GPB
  sensor-group 3
    path sys/intf depth unbounded

What is the correct configuration on nxos?

Looking for support for N9k

Is N9k support 'in the pipeline'?

Call has possible formatting directives when testing in go 1.12.12

FR: Support for arm64 and native macOS platforms

executing pipeline on RPI or macOS fails with the following command:
-bash: ./pipeline: cannot execute binary file: Exec format error
per file:
file pipeline
pipeline: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=2d40fd32ea771a581b43235288815cc9fa3655fe,

Works fine in x86-64 architecture, which is an Ubuntu 18 VM on a Mac mini

Prometheus Pushgateway Timestamp Error

The dump file has correctly collected the data. But, while trying to push the metrics to Prometheus using Push Gateway, Prometheus is not populated with data but with push_time_seconds as 0 and push_failure_time_seconds with large values (Example: 1.5932354227576308e+09).

The Prometheus Pushgateway log collects the following error message:

msg="pushed metrics are invalid or inconsistent with existing metrics" method=POST source=10.244.0.0:17937 err="pushed metrics must not have timestamps"

I guess timestamp has to be removed before pushing it to Prometheus Pushgateway but not sure.

Additionally, I am getting the following error in the Pipeline log:

ERRO[2020-06-27 11:45:11.307455] http reply error="unexpected status code 400 while pushing to http://10.13.92.229:9091/metrics/job/telemetry/instance/pipeline1_18391200352046182925" instance=pipeline1 jobName=telemetry pushGWAddress="http://10.13.92.229:9091" pushURL="http://10.13.92.229:9091/metrics/job/telemetry/instance/pipeline1" tag=pipeline

I am stuck in this problem, please help me to get out of it.

how to set multiple Kafka brokers in pipeline.yml

Does anyone teach me how to set multiple Kafka brokers in pipeline.yml? We’d prepared Kafka cluster with three(3) brokers. And want to send these brokers by setting brokers parameter.

In case of set three brokers, the following setting is true or not?

brokers = kafka.example.com, kafka.example2.com, kafka.example3.com

Filter json for kafka output

It could be useful to have an option, just like with influx/prometheus, to filter the needed sensor leafs going towards kafka bus with the metrics.json file. Any suggestion ?

Confusing / incorrect language in pipeline.conf

In pipeline.conf:

#[gRPCDialout]
...
# encap = gpb
# Encapsulation pushed by client: gpb, gpbkv, gpbcompact.
# As of 6.1.1 release, we support gpb (which is a common
# header used to carry compact and k/v gpb), and we default
# to gpb. In older releases, it might be necessary to configure
# compact or k/v.

So is gpb the only encap supported? What does

"is a common header used to carry compact and k/v gpb"

mean? Is gpb kv supported or not?

In older releases, it might be necessary to configure compact or k/v.

There are no official releases on GitHub, and this is the only source code available. If gpbkv is not supported here and is supported in an older release, is it possible to get access to an older release?

---

[inspector]
stage = xport_output
#
# Module type: tap is only supported in xport_output stage currently.
#
type = tap
...
# encoding = json
#
# Options: json_events | gpb, gpb_kv. If format is a binary format, or
# not supported for input encoding in use (gpb if proto is available,
# gpbk/v or JSON), we fall back to hex. Default is JSON

I'm not sure I understand json_events | gpb, gpb_kv. Was that supposed to be json_events | gpb | gpb_kv?
I tried using gpb_kv, it is not a valid choice - the accepted token is gpbkv.

If format is a binary format, or
not supported for input encoding in use (gpb if proto is available,
gpbk/v or JSON), we fall back to hex

gpbkv / JSON are not supported for any input encoding?
Should that have read "gpb if proto is unavailable"?

transport: http2Client.notifyError got notified that the client transport was broken

Hi,

I am getting below error when tried Dialout to Router from Pipeline, can anyone let us know what could be the issue.

In tcpdump I can see tcp syn and ack messages from Server and router.

Wait for ^C to shutdown
2019/11/11 08:34:30 transport: http2Client.notifyError got notified that the client transport was broken write tcp 192.168.32.18:52662->135.128.182:57500: write: connection reset by peer.
2019/11/11 08:34:30 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 135.21.1282:57500: operation was canceled"; Reconnecting to {135.21.128.182:57500 }
2019/11/11 08:34:30 grpc: addrConn.transportMonitor exits due to: grpc: the connection is closing
2019/11/11 08:34:31 transport: http2Client.notifyError got notified that the client transport was broken read tcp 192.168.32.18:52666->135.228.182:57500: read: connection reset by peer.

help with metrics.json for module: Cisco-IOS-XE-bgp-oper:bgp-state-data/neighbors/neighbor

my pipeline + influxdb is working perfect for cpu-utilization module for Cisco IOS XE. I need to make one for moduel Cisco-IOS-XE-bgp-oper. I have the following but it's not working:
[
{
"basepath" : "Cisco-IOS-XE-bgp-oper:bgp-state-data/neighbors/neighbor",
"spec" : {
"fields" : [
{"name":"bgpneighbor", "tag": true},
{"name":"description"}
]
}
}
]

Can someone help me ?

http2Server.HandleStreams failed to read frame: read tcp ... use of closed network connection

Seeing these transport errors on pipeline server when shutting down. Is this related to grpc-go issue grpc/grpc-go#1062

Interrupt, stopping gracefully
Stopping input 'grpcdialout'
Stopping output 'inspector'
2018/10/10 15:13:37 transport: http2Server.HandleStreams failed to read frame: read tcp ...: use of closed network connection
Done

Is it more of a logging issue and not an error per se?

Pipeline Using a lot of RAM, Crashing

I am running into an issue using Pipeline where the process crashes after a while of collecting telemetry data from about 12 Cisco ASR 9006, 9001, and 9901 routers. These routers are streaming 7 different sensor-paths (uptime, cpu, node summary, optics-info, process-info, and generic-counters) every 10 seconds.

Pipeline normally uses about 5-250MB of RAM and has a "scrape duration" of about 12ms. When Pipeline crashes, RAM usage goes to >1GB, and scrape duration goes >1 second. I can either restart the Pipeline process or reboot the host to get Pipeline working again.

Pipeline.log does show some meaningful log messages. I can post that here if it'll help.

Has anyone ran into this sort of issue?

rename main branch to main

Incorrect node name encoding

Output JSON data includes node names for lists that is non-existant in original data (sensor-paths_PIPELINE_EDIT instead of sensor-paths):

                        "sensor-group": {
                            "id": "BGP-VPN-SENSORS",
                            "sensor-paths_PIPELINE_EDIT": [
                                {
                                    "state": "true",
                                    "status-str": "",
                                    "path": "Cisco-IOS-XR-telemetry-model-driven-oper:telemetry-model-driven/subscriptions/subscription"
                                }
                            ],
                            "configured": 1
                        }

when does it support N9k ?

support for Cat9k?

Hi there.
Is it possible to use pipeline with cat9k device?
Its not an issue though but i am thinking of using cat9k and pipeline, so any comments on this will be highly helpful.

Thanks in advance.

error log when receive telemetry data from nexus9k 7.0(3)I6(1)

I'm using latest pipeline to receive telemetry data from nexus 9k (7.0(3)I6(1)).

the N9K configuration:
telemetry
destination-group 100
ip address 10.140.0.123 port 57500 protocol gRPC encoding GPB
sensor-group 100
path sys/cdp depth 0
subscription 100
dst-grp 100
snsr-grp 100 sample-interval 5000

the configuration file of pipeline is:
[wx_output]
stage = xport_output
type = tap
file = /home/tom/work/data/dump.txt
encoding = json
countonly = false

[n9k_dialout_grpc]
stage = xport_input
type = grpc
encap = gpb
listen = 192.168.156.153:57500
tls = false
logdata = on

Due to I'm running pipeline in a VM, so the internal ip address 192.168.156.153:57500 will be NATed to 10.140.0.123:57500. When pipeline received data from N9K, it report following errors:

INFO[2017-06-30 01:10:35.487100] Conductor says hello, loading config config=wx.conf debug=true fluentd= logfile=pipeline.log maxthreads=2 tag=pipeline.ubuntu version="v1.0.0(bigmuddy)"
DEBU[2017-06-30 01:10:35.487838] Conductor processing section... name=conductor section="wx_output" tag=pipeline.ubuntu
DEBU[2017-06-30 01:10:35.487873] Conductor processing section, type... name=conductor section="wx_output" tag=pipeline.ubuntu type=tap
INFO[2017-06-30 01:10:35.487887] Conductor starting up section name=conductor section="wx_output" stage="xport_output" tag=pipeline.ubuntu
DEBU[2017-06-30 01:10:35.487913] Conductor processing section... name=conductor section="n9k_dialout_grpc" tag=pipeline.ubuntu
DEBU[2017-06-30 01:10:35.487926] Conductor processing section, type... name=conductor section="n9k_dialout_grpc" tag=pipeline.ubuntu type=grpc
INFO[2017-06-30 01:10:35.487939] Conductor starting up section name=conductor section="n9k_dialout_grpc" stage="xport_input" tag=pipeline.ubuntu
INFO[2017-06-30 01:10:35.488009] Metamonitoring: not enabled tag=pipeline.ubuntu
DEBU[2017-06-30 01:10:35.488072] Conductor watching for shutdown... config=wx.conf debug=true logfile=pipeline.log tag=pipeline.ubuntu
INFO[2017-06-30 01:10:35.488154] Starting up tap countonly=false filename="/home/tom/work/data/dump.txt" name="wx_output" streamSpec=&{2 } tag=pipeline.ubuntu
INFO[2017-06-30 01:10:35.488218] gRPC starting block encap=gpb name="n9k_dialout_grpc" server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"
INFO[2017-06-30 01:10:35.488582] gRPC: Start accepting dialout sessions encap=gpb name="n9k_dialout_grpc" server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"

INFO[2017-06-30 01:13:55.069815] gRPC: Receiving dialout stream encap=gpb name="n9k_dialout_grpc" peer="10.75.195.101:37295" server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"
DEBU[2017-06-30 01:13:55.117430] gRPC server logdata encap=gpb msg="00000000 0a 02 73 31 32 07 73 79 73 2f 63 64 70 40 01 5a |..s12.sys/[email protected]|\n00000010 a5 02 7a 1a 12 04 6b 65 79 73 7a 12 12 07 73 79 |..z...keysz...sy|\n00000020 73 2f 63 64 70 2a 07 73 79 73 2f 63 64 70 7a 86 |s/cdp*.sys/cdpz.|\n00000030 02 12 07 63 6f 6e 74 65 6e 74 7a fa 01 7a f7 01 |...contentz..z..|\n00000040 12 09 63 64 70 45 6e 74 69 74 79 7a e9 01 7a e6 |..cdpEntityz..z.|\n00000050 01 12 0a 61 74 74 72 69 62 75 74 65 73 7a d7 01 |...attributesz..|\n00000060 7a 12 12 07 61 64 6d 69 6e 53 74 2a 07 65 6e 61 |z...adminSt*.ena|\n00000070 62 6c 65 64 7a 10 12 0b 63 68 69 6c 64 41 63 74 |bledz...childAct|\n00000080 69 6f 6e 2a 01 20 7a 0d 12 02 64 6e 2a 07 73 79 |ion*. z...dn*.sy|\n00000090 73 2f 63 64 70 7a 26 12 05 6d 6f 64 54 73 2a 1d |s/cdpz&..modTs*.|\n000000a0 32 30 31 37 2d 30 36 2d 32 37 54 32 33 3a 30 30 |2017-06-27T23:00|\n000000b0 3a 34 35 2e 31 35 39 2b 30 30 3a 30 30 7a 25 12 |:45.159+00:00z%.|\n000000c0 08 6d 6f 6e 50 6f 6c 44 6e 2a 19 75 6e 69 2f 66 |.monPolDn*.uni/f|\n000000d0 61 62 72 69 63 2f 6d 6f 6e 66 61 62 2d 64 65 66 |abric/monfab-def|\n000000e0 61 75 6c 74 7a 09 12 04 6e 61 6d 65 2a 01 20 7a |aultz...name*. z|\n000000f0 0c 12 07 6f 70 65 72 45 72 72 2a 01 20 7a 11 12 |...operErr*. z..|\n00000100 06 6f 70 65 72 53 74 2a 07 65 6e 61 62 6c 65 64 |.operSt*.enabled|\n00000110 7a 1a 12 12 70 65 72 73 69 73 74 65 6e 74 4f 6e |z...persistentOn|\n00000120 52 65 6c 6f 61 64 2a 04 74 72 75 65 7a 09 12 02 |Reload*.truez...|\n00000130 72 6e 2a 03 63 64 70 |rn*.cdp|\n" name="n9k_dialout_grpc" peer="10.75.195.101:37295" reqID=0 server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"
ERRO[2017-06-30 01:13:55.118415] gRPC: session closed encap=gpb error=EOF name="n9k_dialout_grpc" server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"

INFO[2017-06-30 01:13:59.716359] gRPC: Receiving dialout stream encap=gpb name="n9k_dialout_grpc" peer="10.75.195.101:37295" server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"
DEBU[2017-06-30 01:13:59.716524] gRPC server logdata encap=gpb msg="00000000 0a 02 73 31 32 07 73 79 73 2f 63 64 70 40 02 5a |..s12.sys/[email protected]|\n00000010 a5 02 7a 1a 12 04 6b 65 79 73 7a 12 12 07 73 79 |..z...keysz...sy|\n00000020 73 2f 63 64 70 2a 07 73 79 73 2f 63 64 70 7a 86 |s/cdp*.sys/cdpz.|\n00000030 02 12 07 63 6f 6e 74 65 6e 74 7a fa 01 7a f7 01 |...contentz..z..|\n00000040 12 09 63 64 70 45 6e 74 69 74 79 7a e9 01 7a e6 |..cdpEntityz..z.|\n00000050 01 12 0a 61 74 74 72 69 62 75 74 65 73 7a d7 01 |...attributesz..|\n00000060 7a 12 12 07 61 64 6d 69 6e 53 74 2a 07 65 6e 61 |z...adminSt*.ena|\n00000070 62 6c 65 64 7a 10 12 0b 63 68 69 6c 64 41 63 74 |bledz...childAct|\n00000080 69 6f 6e 2a 01 20 7a 0d 12 02 64 6e 2a 07 73 79 |ion*. z...dn*.sy|\n00000090 73 2f 63 64 70 7a 26 12 05 6d 6f 64 54 73 2a 1d |s/cdpz&..modTs*.|\n000000a0 32 30 31 37 2d 30 36 2d 32 37 54 32 33 3a 30 30 |2017-06-27T23:00|\n000000b0 3a 34 35 2e 31 35 39 2b 30 30 3a 30 30 7a 25 12 |:45.159+00:00z%.|\n000000c0 08 6d 6f 6e 50 6f 6c 44 6e 2a 19 75 6e 69 2f 66 |.monPolDn*.uni/f|\n000000d0 61 62 72 69 63 2f 6d 6f 6e 66 61 62 2d 64 65 66 |abric/monfab-def|\n000000e0 61 75 6c 74 7a 09 12 04 6e 61 6d 65 2a 01 20 7a |aultz...name*. z|\n000000f0 0c 12 07 6f 70 65 72 45 72 72 2a 01 20 7a 11 12 |...operErr*. z..|\n00000100 06 6f 70 65 72 53 74 2a 07 65 6e 61 62 6c 65 64 |.operSt*.enabled|\n00000110 7a 1a 12 12 70 65 72 73 69 73 74 65 6e 74 4f 6e |z...persistentOn|\n00000120 52 65 6c 6f 61 64 2a 04 74 72 75 65 7a 09 12 02 |Reload*.truez...|\n00000130 72 6e 2a 03 63 64 70 |rn*.cdp|\n" name="n9k_dialout_grpc" peer="10.75.195.101:37295" reqID=0 server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"
ERRO[2017-06-30 01:13:59.716713] gRPC: session closed encap=gpb error=EOF name="n9k_dialout_grpc" server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"
INFO[2017-06-30 01:14:04.392640] gRPC: gRPC client loop, rxed SHUTDOWN, closing connections ctrl_msg_id=0 encap=gpb name="n9k_dialout_grpc" server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"
DEBU[2017-06-30 01:14:04.392718] gRPC: waiting for children encap=gpb name="n9k_dialout_grpc" server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"
INFO[2017-06-30 01:14:04.392737] gRPC: Stop accepting dialout sessions encap=gpb name="n9k_dialout_grpc" server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"
DEBU[2017-06-30 01:14:04.392764] gRPC: gRPC server notify conductor binding is closed encap=gpb name="n9k_dialout_grpc" server=192.168.156.153:57500 tag=pipeline.ubuntu type="pipeline is SERVER"
INFO[2017-06-30 01:14:04.392783] tap feeder loop, rxed SHUTDOWN countonly=false filename="/home/tom/work/data/dump.txt" name="wx_output" streamSpec=&{2 } tag=pipeline.ubuntu
INFO[2017-06-30 01:14:04.392811] Goodbye tag=pipeline.ubuntu

Error with go get

I get this error when I try to build.
Centos 7.4.1708
Go 1.9.3
package github.com/sirupsen/logrus: case-insensitive import collision: "github.com/sirupsen/logrus" and "github.com/Sirupsen/logrus"

Pipeline dial-in does not recover after device reboot

After a device reboots, the dial-in does not automatically resume once the device is operational again. Behavior observed during device upgrade to IOS XR 6.3.2. Pipeline logs do not indicate there is any issue during period of device upgrade.

Does Pipeline Replay preserve timestamps?

Does the Pipeline Replay feature preserve the timestamps of messages processed?

For instance - when exporting in to InfluxDB, will the data appear differently based on inter-message delay? Or is the inter-message delay purely for scaling processing workload over time?

PIpeline is not dumping into influxdb.

HI
I am receiving telemetry data but pipeline is discarding it only for voltages.
below is the metrics2.json extract i am using for pipeline.However temperature gets into influxdb.
Both temperature and voltages has same yang models.
Any idea whats is wrong here?

{
	"basepath" : "Cisco-IOS-XR-sysadmin-envmon-ui:envmon/environment/oper/temperatures/location/sensor_attributes",
	"spec" : {
		"fields" : [
			{"name" : "location", "tag" : true},
			{"name" : "sensor"},
			{"name" : "sensor_id"},
			{"name" : "alarm"},
			{"name" : "temperature_value"},
			{"name" : "value"},
			{"name" : "critical_lo"},
			{"name" : "major_lo"},
			{"name" : "minor_lo"},
			{"name" : "major_hi"},
			{"name" : "minor_hi"},
			{"name" : "critical_hi"}
		]
	}
},
    {
	"basepath" : "Cisco-IOS-XR-sysadmin-envmon-ui:envmon/environment/oper/voltages/location/sensor_attributes",
	"spec" : {
		"fields" : [
			{"name" : "location", "tag" : true},
			{"name" : "sensor"},
			{"name" : "sensor_id"},
			{"name" : "alarm"},
                            {"name" : "value"},
			{"name" : "temperature_value"},
			{"name" : "critical_lo"},
			{"name" : "major_lo"},
			{"name" : "minor_lo"},
			{"name" : "major_hi"},
			{"name" : "minor_hi"},
			{"name" : "critical_hi"}
		]
	}
},

Thanks,
NIthin

Failed to extract next buffer

Any idea what the following pipeline log could be for and how can I resolve it?

time="2019-08-30 07:49:13.386626" level=error msg="Failed to extract next buffer" error="ENCAP ST: nextBlockBuffer failed; msg too long [1228565]" fields.msg="00000000 00 01 00 01 00 01 00 00 00 12 bf 15 |............|\n" len=12 tag=pipeline1