narmi / cloudwatchmetricbeat Goto Github PK
View Code? Open in Web Editor NEWA beat for AWS CloudWatch Metrics. Located in NYC? We are hiring: https://www.narmi.com/careers/
License: Other
A beat for AWS CloudWatch Metrics. Located in NYC? We are hiring: https://www.narmi.com/careers/
License: Other
Could you please help me on this. Cannot use beat properly. It fails after first request.
Thanks in advance,
Leonid
root@elk:~/go/src/github.com/narmitech/cloudwatchmetricbeat# cloudwatchmetricbeat -c cloudwatchmetricbeat.yml -e -d "*"
2018/05/14 14:26:16.042086 beat.go:285: INFO Home path: [/root/go/src/github.com/narmitech/cloudwatchmetricbeat] Config path: [/root/go/src/github.com/narmitech/cloudwatchmetricbeat] Data path: [/root/go/src/github.com/narmitech/cloudwatchmetricbeat/data] Logs path: [/root/go/src/github.com/narmitech/cloudwatchmetricbeat/logs]
2018/05/14 14:26:16.044414 beat.go:186: INFO Setup Beat: cloudwatchmetricbeat; Version: 5.3.4
2018/05/14 14:26:16.044429 processor.go:44: DBG Processors:
2018/05/14 14:26:16.044442 beat.go:192: DBG Initializing output plugins
2018/05/14 14:26:16.044509 output.go:254: INFO Loading template enabled. Reading template file: /root/go/src/github.com/narmitech/cloudwatchmetricbeat/cloudwatchmetricbeat.template.json
2018/05/14 14:26:16.044650 output.go:265: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /root/go/src/github.com/narmitech/cloudwatchmetricbeat/cloudwatchmetricbeat.template-es2x.json
2018/05/14 14:26:16.044796 client.go:123: INFO Elasticsearch url: http://localhost:9200
2018/05/14 14:26:16.044828 outputs.go:108: INFO Activated elasticsearch as output plugin.
2018/05/14 14:26:16.044848 publish.go:238: DBG Create output worker
2018/05/14 14:26:16.044886 publish.go:280: DBG No output is defined to store the topology. The server fields might not be filled.
2018/05/14 14:26:16.044909 publish.go:295: INFO Publisher name: elk
2018/05/14 14:26:16.045031 async.go:63: INFO Flush Interval set to: 1s
2018/05/14 14:26:16.045050 async.go:64: INFO Max Bulk Size set to: 50
2018/05/14 14:26:16.045058 async.go:72: DBG create bulk processing worker (interval=1s, bulk size=50)
2018/05/14 14:26:16.045268 cloudwatchmetricbeat.go:44: INFO settings: config={Period:1m0s AWSRegion:us-east-1 Prospectors:[{Id:billing Metrics:[{AWSStatistics:[Maximum] AWSDimensions:[Currency] AWSDimensionSelect:map[Currency:[USD]] AWSDimensionsSelectParam:map[] AWSNamespace:AWS/Billing AWSMetricName:EstimatedCharges RangeSeconds:86400 PeriodSeconds:0 DelaySeconds:0}]} {Id:ec2_cloudwatch Metrics:[{AWSStatistics:[Average] AWSDimensions:[InstanceId] AWSDimensionSelect:map[InstanceId:[id-09005aa854cb13b23]] AWSDimensionsSelectParam:map[] AWSNamespace:AWS/EC2 AWSMetricName:CPUUtilization RangeSeconds:0 PeriodSeconds:0 DelaySeconds:0} {AWSStatistics:[Average] AWSDimensions:[InstanceId] AWSDimensionSelect:map[InstanceId:[id-09005aa854cb13b23]] AWSDimensionsSelectParam:map[] AWSNamespace:AWS/EC2 AWSMetricName:NetworkOut RangeSeconds:0 PeriodSeconds:0 DelaySeconds:0}]} {Id:vpn_mon Metrics:[{AWSStatistics:[Average] AWSDimensions:[VpnId] AWSDimensionSelect:map[VpnId:[vpn-61687a11]] AWSDimensionsSelectParam:map[] AWSNamespace:AWS/VPN AWSMetricName:TunnelState RangeSeconds:3600 PeriodSeconds:0 DelaySeconds:0}]}]}
2018/05/14 14:26:16.045381 beat.go:221: INFO cloudwatchmetricbeat start running.
2018/05/14 14:26:16.045392 cloudwatchmetricbeat.go:63: INFO cloudwatchmetricbeat is running! Hit CTRL-C to stop it.
2018/05/14 14:26:16.045424 metrics.go:23: INFO Metrics logging every 30s
2018/05/14 14:26:46.045680 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/05/14 14:27:16.045624 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/05/14 14:27:16.175219 cloudwatchmetricbeat.go:194: INFO making request: params={
Dimensions: [{
Name: "VpnId",
Value: "vpn-61687a11"
}],
EndTime: 2018-05-14 14:22:16.045654982 +0000 UTC,
MetricName: "TunnelState",
Namespace: "AWS/VPN",
Period: 60,
StartTime: 2018-05-14 13:22:16.045654982 +0000 UTC,
Statistics: ["Average"]
}
2018/05/14 14:27:16.175281 metrics.go:51: INFO Total non-zero values:
2018/05/14 14:27:16.175293 metrics.go:52: INFO Uptime: 1m0.136432267s
2018/05/14 14:27:16.175301 beat.go:225: INFO cloudwatchmetricbeat stopped.
I think the code has a bug:
$:/go/src/github.com/narmitech/cloudwatchmetricbeat$ pwd/go/src/github.com/narmitech/cloudwatchmetricbeat$ make
/home/jasonlang/go/src/github.com/narmitech/cloudwatchmetricbeat
$:
go build -i -ldflags "-X github.com/elastic/beats/libbeat/version.buildTime=2018-06-20T17:26:32Z -X github.com/elastic/beats/libbeat/version.commit=fbf0b84fb01fd271b26122dee758166b20e226ca"
beater/cloudwatchmetricbeat.go:25:9: undefined: publisher.Client
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:112: recipe for target 'cloudwatchmetricbeat' failed
make: *** [cloudwatchmetricbeat] Error 2
I'm running Docker version 18.06.1-ce, build e68fc7a. When running the Dockerfile, the following copy statement isn't recursive, and appears to only transfer the Dockerfile itself albeit to the correct directory.
COPY . $GOPATH/src/github.com/narmitech/cloudwatchmetricbeat
This is most likely a Docker bug I'd bet, but just to be aware there's a problem right now.
Something Odd is going on when I attempt to use the beat to pull RDS metrics from cloudwatch. It's not using the correct Dimension.
Config:
prospectors: - id: rds-metrics metrics: - aws_namespace: "AWS/RDS" aws_dimensions: [DBClusterIdentifier] aws_dimensions_select: InstanceId: [aurora1-cluster] aws_metric_name: CPUUtilization aws_statistics: [Average]
Log:
2018-07-31T17:42:28Z INFO making request: params={ Dimensions: [{ Name: "InstanceId", Value: "aurora1-cluster" }], EndTime: 2018-07-31 17:37:28.871505767 +0000 UTC m=-119.993549520, MetricName: "CPUUtilization", Namespace: "AWS/RDS", Period: 60, StartTime: 2018-07-31 17:27:28.871505767 +0000 UTC m=-719.993549520, Statistics: ["Average"]
Do you have default Kibana dashboards for this beat?
Guys,
Any plans to make it compatible with elastic/beats version 7.0 (alpha) or latest 6.* at least?
Would it be possible to add support for subj. to have something like:
aws_dimensions: [InstanceId]
aws_dimensions_select:
InstanceId: [i-abcd, i-ifgh,]
aws_statistics: [Average]
So that the beat can be run in a way that is not dependent on one machine, we should track the status of each metric's scraping in a registry. This could also be used to backfill data on the first run if no previous timestamp is present.
This could be similar to https://github.com/e-travel/cloudwatchlogsbeat/blob/fe67ee1cfe06e08cc1338ceb1417efc32efa5d9a/beater/s3.go
can you please installation steps.. clearly...
i am getting below error..
path :- /tmp/cloudwatchmetricbeat/beater
=> go build
# _/tmp/cloudwatchmetricbeat/beater
Error:-
./cloudwatchmetricbeat.go:25:9: undefined: publisher.Client
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.