Hello,

I have a infinite loop problem when running httpbeat. It occurs randomly, but faster (10-15sec) using logstash output. With file output it seem to run longer (3-4 min) but ultimatly start to loop. The CPU is then stuck at 100% User.

I've seen that on libbeat that may be related : elastic/beats#878

Maybe you can update the lib version in dependencies ? I'm new to Go and I'm not sure how to do this.

I'm on debian Jessie, up to date, Go 1.5.1 x64

Here is my test config

httpbeat:
  urls:
    -
      cron: "@every 5s"
      url: http://es-03:9200/_cluster/stats
      method: get
      document_type: httpbeat_json
      headers:
      fields:
output:
  file:
    path: "/tmp/filebeat"
    filename: filebeat
    rotate_every_kb: 10000
    number_of_files: 7
  console:
    pretty: true
shipper:
logging:
  to_syslog: true
  files:
    rotateeverybytes: 10485760 # = 10MB

A strace doesn't help at all, the main process is stuck on a system futex call and the third thread is in pure user mode so strace doesn't help here.

Here is the lsof output. I'm testing HTTPBeat monitoring my elasticsearch cluster (es:9200/_cluster/stats) so the es-03.my.tld are connections to ES by httpbeat for checks. I'm not using httpbeat ES output. I'm pretty surprised to see more that one HTTP connection to the es-03 server since there is only one check at a time. Maybe a non-closed socket here ?

COMMAND    PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
httpbeat 16076 root  cwd    DIR              202,1     4096  434641 /srv/go/src/github.com/christiangalsterer/httpbeat/etc
httpbeat 16076 root  rtd    DIR              202,1     4096       2 /
httpbeat 16076 root  txt    REG              202,1 10931624  285150 /srv/go/src/github.com/christiangalsterer/httpbeat/httpbeat
httpbeat 16076 root  mem    REG              202,1  1738176  268580 /lib/x86_64-linux-gnu/libc-2.19.so
httpbeat 16076 root  mem    REG              202,1   137440  268576 /lib/x86_64-linux-gnu/libpthread-2.19.so
httpbeat 16076 root  mem    REG              202,1   140928  268577 /lib/x86_64-linux-gnu/ld-2.19.so
httpbeat 16076 root    0u   CHR              136,2      0t0       5 /dev/pts/2
httpbeat 16076 root    1u   CHR              136,2      0t0       5 /dev/pts/2
httpbeat 16076 root    2u   CHR              136,2      0t0       5 /dev/pts/2
httpbeat 16076 root    3u  unix 0xffff880204551440      0t0 4378679 socket
httpbeat 16076 root    4u  0000                0,9        0    6706 anon_inode
httpbeat 16076 root    5u  unix 0xffff880204551b40      0t0 4378680 socket
httpbeat 16076 root    6u  unix 0xffff88005b610140      0t0 4378681 socket
httpbeat 16076 root    7u  unix 0xffff88005b3e13c0      0t0 4378682 socket
httpbeat 16076 root    8u  unix 0xffff88005b3e1ac0      0t0 4378683 socket
httpbeat 16076 root    9u  unix 0xffff8800ea7a6180      0t0 4378684 socket
httpbeat 16076 root   10u  unix 0xffff8800ea7a6500      0t0 4378685 socket
httpbeat 16076 root   11u  unix 0xffff880204931780      0t0 4378686 socket
httpbeat 16076 root   12u  IPv4            4380277      0t0     TCP 10.0.42.237:47968->es-03.my.tld:9200 (ESTABLISHED)
httpbeat 16076 root   13u   REG              202,1   141967  285129 /tmp/filebeat/filebeat
httpbeat 16076 root   14u  IPv4            4382273      0t0     TCP 10.0.42.237:47981->es-03.my.tld:9200 (ESTABLISHED)
httpbeat 16076 root   15u  IPv4            4382328      0t0     TCP 10.0.42.237:47992->es-03.my.tld:9200 (ESTABLISHED)
httpbeat 16076 root   17u  IPv4            4380033      0t0     TCP 10.0.42.237:47851->es-03.my.tld:9200 (ESTABLISHED)
httpbeat 16076 root   18u  IPv4            4380062      0t0     TCP 10.0.42.237:47865->es-03.my.tld:9200 (ESTABLISHED)
httpbeat 16076 root   19u  IPv4            4380089      0t0     TCP 10.0.42.237:47878->es-03.my.tld:9200 (ESTABLISHED)
httpbeat 16076 root   20u  IPv4            4380112      0t0     TCP 10.0.42.237:47889->es-03.my.tld:9200 (ESTABLISHED)
httpbeat 16076 root   21u  IPv4            4380143      0t0     TCP 10.0.42.237:47904->es-03.my.tld:9200 (ESTABLISHED)
httpbeat 16076 root   22u  IPv4            4380166      0t0     TCP 10.0.42.237:47915->es-03.my.tld:9200 (ESTABLISHED)
httpbeat 16076 root   23u  IPv4            4380197      0t0     TCP 10.0.42.237:47930->es-03.my.tld:9200 (ESTABLISHED)
httpbeat 16076 root   24u  IPv4            4380220      0t0     TCP 10.0.42.237:47941->es-03.my.tld:9200 (ESTABLISHED)
httpbeat 16076 root   25u  IPv4            4380248      0t0     TCP 10.0.42.237:47954->es-03.my.tld:9200 (ESTABLISHED)

endpoint provides:
"values":[265381,276756,16476547,42900243,44762036, ...

httpbeat pushes:

"values": [
265381,
276756,
1.6476547e+07,
4.2900243e+07,
4.4762036e+07,

this leads to an error when using graylog as logstash destination:
[12]: index [graylog2_17], type [message], id [e6449a00-ae49-11e6-bf3c-5e4abd8a540e], message [MapperParsingException[failed to parse]; nested: IllegalArgumentException[mapper [beat_response_jsonBody_snapshot_values] of different type, current_type [long], merged_type [double]];]

potential solution in:
http://stackoverflow.com/questions/22343083/json-marshaling-with-long-numbers-in-golang-gives-floating-point-number

Hi,

Is there a way to put the current date in the url and/or the body of a request ?
i.e:
urls:
-
url: http://localhost:9200/logevent-%{+yyyy.MM.dd}

We use httpbeat to query data from multiple servers. It would be useful to have a feature where the http queries can be added to the httpbeat.yml and Http beat dynamically reloads the new configuration without requiring a restart. All official Elastic beats support this feature.

Hello,

I am attempting to recreate HTTPS request. When we issue curl request it looks like this...
curl --tlsv1.2 -n -k https://hostname:port/restendpoint

I configured httpbeat.yml, see below

   basic_auth:
     username: username
     password: password
#Optional ssl configuration. By default is off.
   ssl:
     #List of root certificates for HTTPS server verifications
     #certificate_authorities: ["/etc/pki/root/ca.pem"]

     #Certificate for TLS client authentication
     #certificate: "/etc/pki/client/cert.pem"

     # Client Certificate Key
     #key: "/etc/pki/client/cert.key"

     #This option controls whether the client verifies server certificates and host
     #names. The values `none` and `full` can be used. If `verification_mode` is set
     #to `none`, all server host names and certificates are accepted. In this mode,
     #TLS-based connections are susceptible to man-in-the-middle attacks. Use this
     #option for testing only.

     #The default is `full`.
     verification_mode: none

     #Configure cipher suites to be used for TLS connections
     #cipher_suites: []

     #Configure curve types for ECDHE based cipher suites
     #curve_types: []

     #List of allowed SSL/TLS versions. If SSL/TLS server decides for protocol versions
     #not configured, the connection will be dropped during or after the handshake. The
     #setting is a list of allowed protocol versions:
     #`SSLv3`, `TLSv1` for TLS version 1.0, `TLSv1.0`, `TLSv1.1` and `TLSv1.2`.

     #The default value is `[TLSv1.0, TLSv1.1, TLSv1.2]`.
     versions: [TLSv1.2]

I am sending directly to elasticsearch and also output to file but no data has being written to either.

I only get this INFO message in log.
2017/01/06 19:56:03.162631 logp.go:232: INFO No non-zero metrics in the last 30s

Any help would be appreciated!
Thanks

Hi,

We're using httpbeat 4.0.0 to monitor a few URLs.
After a few days of running, httpbeat crashes with this error message:

An error occurred while executing HTTP request: [No method specified]

What is the problem?

Regards,
Eyal

I suppose this is more of a feature request or I guess it could be considered documentation request if this already exists... but I'd like to be able to use your beat to input JSON arrays, for whatever reason my body message comes in as a JSON string instead of actually getting unmarshalled as JSON, this appears to be an issue with elasticsearch rather than your beat since I have to wrap my json array in an object before I can even curl it into ES manually. So ideally adding a "field" under JSONBODY would be the ideal way to do this, or perhaps that already exists and I'm just not doing it right, either way any help would be appreciated. I'm more than happy to send you a pull request if I can understand where / how this is being done.

Hi,
This is more of a question ;)! Just wondering if there is something equivalent to config_dir in filebeat where we can put additional configuration files. I think it's useful when we've multiple hosts to monitor, it would be nice to save the conf of each host in its own file.

Thank you

Hi,

I'm trying to use 'httpbeat' in order to sample a spring boot application mertics.
The url I'm sampling is: http://localhost:8080/metrics
The output format is of type: 'json'

The response contains the following JSON:

{"mem":824349,"mem.free":222040,"processors":8,"instance.uptime":11355545,"uptime":9690309,"systemload.average":4.31494140625,"heap.committed":748032,"heap.init":262144,"heap.used":525991,"heap":3728384,"nonheap.committed":78144,"nonheap.init":2496,"nonheap.used":76320,"nonheap":0,"threads.peak":56,"threads.daemon":33,"threads.totalStarted":1398,"threads":37,"classes":8172,"classes.loaded":8242,"classes.unloaded":70,"gc.ps_scavenge.count":14,"gc.ps_scavenge.time":132,"gc.ps_marksweep.count":4,"gc.ps_marksweep.time":612,"counter.status.200.health":343,"counter.status.200.metrics.root":2,"counter.status.200.metrics.name:.-star-":107,"gauge.response.star-star.favicon.ico":19.0,"counter.status.200.star-star.favicon.ico":1,"gauge.response.metrics":2.0,"gauge.response.metrics.root":2.0,"gauge.response.metrics.name:.-star-":1.0,"gauge.response.health":2.0,"counter.status.200.metrics":879,"httpsessions.max":-1,"httpsessions.active":0}

You can see that some of the values in the response contain a dot '.'

While 'httpbeat' tring to send the response to elasticSearch I got the following error:

2017/01/19 16:41:36.714328 client.go:432: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"Field name [mem.free] cannot contain '.'"}

I assume that I can change the mapping in httpbeat.template.json to make elasticsearch tollerant for this naming confession.

Is it true?
If yes - can you please advise how should I change the mapping?

Thanks!

Trying to launch httpbeat with elasticsearch plugin as the output, facing an issue -

• Ran the XPUT template command first to create index in elasticsearch - success
• Modified Httpbeat.yml file to send output to elasticsearch
• Start Httpbeat with the command below
  ./httpbeat -c httpbeat.yml -e -d "*"
  Error msg -
  ERR failed to initialize elasticsearch plugin as output: Error loading template /usr/local/go/src/github.com/christiangalsterer/httpbeat/httpbeat.template-es2x.json: open /usr/local/go/src/github.com/christiangalsterer/httpbeat/httpbeat.template-es2x.json: no such file or directory

/v2/meters/cpu_util?q.field=resource_id&q.op=eq&q.value=d9191b5a-be7a-478e-b31a-cfeef3e3562d&q.field=timestamp&q.op=ge&q.value=2017-06-15T02:05:33
&q.field=timestamp&q.op=le&q.value=2017-06-15T12:05:33,
At httpbeat.yml's config,this time like 2017-06-15T02:05:33 should how to do? thank you so much!!!!

Hi,

We're using httpbeat to sample every minute (cron: "@every 1m") multiple URLs from http://127.0.0.1:8443 on a Windows 2012 R2 machine.
It works great for a list of a few URLs, but when we increase the list to around 30 URLs or more, we sometimes get this error in the LOG file:

<date_and_time> ERR An error occurred while executing HTTP request: [dial tcp 127.0.0.1:8443: connectex: No connection could be made because the target machine actively refused it. Get http://127.0.0.1:8443/<rest_of_URL>: dial tcp 127.0.0.1:8443: connectex: No connection could be made because the target machine actively refused it.]

It isn't a problem on the web server side, because the same sample of URLs with the same frequency works just fine using a different sampling tool.

Is there a limit to the number of URLs we can sample with one instance of httpbeat, or did we find a performance problem?

Please advise,
Eyal

Currently working around by mutating on the logstash side, but I would prefer if httpbeat would simply not send a certain request header.

I ran into the following error after cloning from github.

make

first make sure we have godep

go get github.com/tools/godep
/Users/stevesui/go_work:/Users/stevesui/go_work/httpbeat/bin/godep go build
make: /Users/stevesui/go_work:/Users/stevesui/go_work/httpbeat/bin/godep: No such file or directory
make: *** [httpbeat] Error 1

Basically, I am not sure when the bin directory will be created during the make process.

Thanks.

In addition to the HTTP status code, also the status phrase should be returned, similar as in the Http protocol module in Packetbeat.

Beats 5.0.0 has been released; it looks like httpbeat needs some work to build against it:

$ make
go build
# _/usr/src/httpbeat
./main.go:13: cannot use beater.New (type func(*"github.com/christiangalsterer/httpbeat/vendor/github.com/elastic/beats/libbeat/beat".Beat, *"github.com/christiangalsterer/httpbeat/vendor/github.com/elastic/beats/libbeat/common".Config) ("github.com/christiangalsterer/httpbeat/vendor/github.com/elastic/beats/libbeat/beat".Beater, error)) as type "github.com/elastic/beats/libbeat/beat".Creator in argument to "github.com/elastic/beats/libbeat/beat".Run
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:73: recipe for target 'httpbeat' failed
make: *** [httpbeat] Error 2

Any chance we can get an updated httpbeat for beats 5.0.0 ?

When there is no load on machine with httb beat every thing is ok, when we run load test, httpbeat begin consume more resources than our application. What is the reason if it still just should pull links and send responses to logstash?

Consumed memory:

Configuration:
`
urls:
# Each - URL endpoints to call. Below are the URL endpoint specific configurations
#Url to monitor
-
url: http://localhost:80/service1/metrics
method: get
cron: '@every 15s'
# Optional additional headers to send to the endpoint
headers:
Accept: application/json
output_format: json
json_dot_mode: replace
#Url to monitor
-
url: http://localhost:80/service2/metrics
method: get
cron: '@every 15s'
# Optional additional headers to send to the endpoint
headers:
Accept: application/json
output_format: json
json_dot_mode: replace
#Url to monitor
-
url: http://localhost:80/service3/metrics
method: get
cron: '@every 15s'
# Optional additional headers to send to the endpoint
headers:
Accept: application/json
output_format: json
json_dot_mode: replace
#Url to monitor
-
url: http://localhost:80/service4/metrics
method: get
cron: '@every 15s'
# Optional additional headers to send to the endpoint
headers:
Accept: application/json
output_format: json
json_dot_mode: replace

`

Hi,

Is it possible to include the response/transaction time?

Thanks

Thanks @christiangalsterer ，
your work helps me a lot, while I was trying to use the httpbeat. First I install gvm,then I install golang1.4 and golang1.6.3,then glide,then I cd $GOROOT/src and I git clone httpbeat,after I make httpbeat ,these errors appears, can you help me? thanks
main.go:5:2: cannot find package "github.com/christiangalsterer/httpbeat/beat" in any of:
/home/ubuntu/.gvm/gos/go1.6.3/src/httpbeat/vendor/github.com/christiangalsterer/httpbeat/beat (vendor tree)
/home/ubuntu/.gvm/gos/go1.6.3/src/vendor/github.com/christiangalsterer/httpbeat/beat
/home/ubuntu/.gvm/gos/go1.6.3/src/github.com/christiangalsterer/httpbeat/beat (from $GOROOT)
/home/ubuntu/.gvm/pkgsets/go1.6.3/global/src/github.com/christiangalsterer/httpbeat/beat (from $GOPATH)
make: *** [httpbeat] Error 1

I might be way off, but I checked out master and every release you tagged going back to 1.0.0 and I pretty much get the same error... what am I missing here, and could you please add it to your documentation?

make httpbeat
go build
# _/Users/dmorga003c/git/httpbeat
./main.go:12: cannot use "github.com/christiangalsterer/httpbeat/beat".New() (type *"github.com/christiangalsterer/httpbeat/beat".Httpbeat) as type "github.com/elastic/beats/libbeat/beat".Creator in argument to "github.com/elastic/beats/libbeat/beat".Run
make: *** [httpbeat] Error 2

The response body is now stored in Elasticsearch as a plain string.
In case of monitoring Spring Boot metrics that are returned as a JSON, there is no way to query these metrics in Elasticsearch and visualize them in Kibana.
Please provide an option to store the JSON response as some nested structure in the database.
There could probably be a parameter in the config file to specify an expected type of the HTTP response: text/json/...

I'm trying to use this to download a json document from a rest endpoint (our RavenDB document store) and it doesn't support basic authentication (only NTLM authentication). I can accomplish this with curl by using the --ntlm flag. Is there a way to do this in httpbeat?

Thanks

Pressing ctrl+c gives an ugly stack trace except a proper shutdown.

I’m currently pulling metrics with httpbeat and sending them to Elasticsearch and visualizing it with kibana. I had issues getting data with kibana to display in charts because the httpbeat is using nested for the actual data type. Kibana doesn’t work great with nested objects (elastic/kibana#1084). I solved the problem by changing the template to use object type instead.
Taking into account that probably most people will use httpbeat with kibana, wouldn’t it give a better experience out of the box if by default the template used object as the type for the data?
If you like the idea I can send you a PR.
Thank you for httpbeat! Great job!

The documentation/templates state the HTTP status code is in the code field, but the name in the event is statusCode.

The release file httpbeat-4.0.0-x86_64.rpm contains a package with version 5.2.1_SNAPSHOT

$> rpm -qip  httpbeat-4.0.0-x86_64.rpm 
Name        : httpbeat                     Relocations: / 
Version     : 5.2.1_SNAPSHOT                    Vendor: Elastic
Release     : 1                             Build Date: dim. 19 févr. 2017 08:31:43 CET
Install Date: (not installed)               Build Host: bb0646f58027
Group       : default                       Source RPM: httpbeat-5.2.1_SNAPSHOT-1.src.rpm
Size        : 29246589                         License: ASL 2.0
Signature   : (none)
Packager    : <@bb0646f58027>
URL         : https://github.com/christiangalsterer/httpbeat
Summary     : Queries arbitrary HTTP endpoints in a regular interval and ships the result to the configured output channel.
Description :
Queries arbitrary HTTP endpoints in a regular interval and ships the result to the configured output channel.```

I couldn't find where this rpm 'dedicated' version is stored in the repo (so I've no PR to fix it)