elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash

Beats - The Lightweight Shippers of the Elastic Stack

The Beats are lightweight data shippers, written in Go, that you install on your servers to capture all sorts of operational data (think of logs, metrics, or network packet data). The Beats send the operational data to Elasticsearch, either directly or via Logstash, so it can be visualized with Kibana.

By "lightweight", we mean that Beats have a small installation footprint, use limited system resources, and have no runtime dependencies.

This repository contains libbeat, our Go framework for creating Beats, and all the officially supported Beats:

Beat	Description
Auditbeat	Collect your Linux audit framework data and monitor the integrity of your files.
Filebeat	Tails and ships log files
Functionbeat	Read and ships events from serverless infrastructure.
Heartbeat	Ping remote services for availability
Metricbeat	Fetches sets of metrics from the operating system and services
Packetbeat	Monitors the network and applications by sniffing packets
Winlogbeat	Fetches and ships Windows Event logs
Osquerybeat	Runs Osquery and manages interraction with it.

In addition to the above Beats, which are officially supported by Elastic, the community has created a set of other Beats that make use of libbeat but live outside of this Github repository. We maintain a list of community Beats here.

Documentation and Getting Started

You can find the documentation and getting started guides for each of the Beats on the elastic.co site:

• Beats platform
• Auditbeat
• Filebeat
• Functionbeat
• Heartbeat
• Metricbeat
• Packetbeat
• Winlogbeat

Documentation and Getting Started information for the Elastic Agent

You can find the documentation and getting started guides for the Elastic Agent on the elastic.co site

Getting Help

If you need help or hit an issue, please start by opening a topic on our discuss forums. Please note that we reserve GitHub tickets for confirmed bugs and enhancement requests.

Downloads

You can download pre-compiled Beats binaries, as well as packages for the supported platforms, from this page.

Contributing

We'd love working with you! You can help make the Beats better in many ways: report issues, help us reproduce issues, fix bugs, add functionality, or even create your own Beat.

Please start by reading our CONTRIBUTING file.

Building Beats from the Source

See our CONTRIBUTING file for information about setting up your dev environment to build Beats from the source.

Snapshots

For testing purposes, we generate snapshot builds that you can find here. Please be aware that these are built on top of main and are not meant for production.

CI

PR Comments

It is possible to trigger some jobs by putting a comment on a GitHub PR. (This service is only available for users affiliated with Elastic and not for open-source contributors.)

• beats
  • jenkins run the tests please or jenkins run tests or /test will kick off a default build.
  • /test macos will kick off a default build with also the macos stages.
  • /test <beat-name> will kick off the default build for the given PR in addition to the <beat-name> build itself.
  • /test <beat-name> for macos will kick off a default build with also the macos stage for the <beat-name>.
• apm-beats-update
  • /run apm-beats-update
• apm-beats-packaging
  • /package or /packaging will kick of a build to generate the packages for beats.
• apm-beats-tester
  • /beats-tester will kick of a build to validate the generated packages.

PR Labels

It's possible to configure the build on a GitHub PR by labelling the PR with the below labels

• <beat-name> to force the following builds to run the stages for the <beat-name>
• macOS to force the following builds to run the macos stages.

beats's Issues

Centos/RHEL/OEL version 5 rpms

Is it possible to get a package made for release 5? Many enterprises are stuck on this version due to application certification issues. Oracle software mostly certified on OEL 5. Is it possible or not due to code features?

No hosts list or country distribution showing

used the deploy script with ansible, had to tinker a bit because of my own issues, but I am not getting any results in the traffic distribution or the hosts list. Can you advise why that might be? Thanks!

[Enhancement] Integration with Zabbix

Hi,

Are there any plans to integrate PacketBeat with Zabbix (http://www.zabbix.com)? Thus sending events like high response times to Zabbix, which in turn will send alerts to the relevant support staff?

Or putting in another way, will PacketBeat get a API in future? Or does it have one now?

Regards,

Dawid

I can't install packetbeat

$rpm -ivh packetbeat-0.1.0-1.el6.x86_64.rpm
error: Failed dependencies:
daemonize is needed by packetbeat-0.1.0-1.el6.x86_64

log_unix.go:113: decodePktEth exception - runtime error: index out of range

Jul  3 11:08:32 int-nginx-02 /usr/bin/packetbeat[20877]: log_unix.go:113: decodePktEth exception. Recovering, but please report this: runtime error: index out of range.
Jul  3 11:08:32 int-nginx-02 /usr/bin/packetbeat[20877]: log_unix.go:114: Stacktrace: /home/vagrant/src/packetbeat/log_unix.go:114 (0x4250ae)#012/usr/local/go/src/pkg/runtime/panic.c:248 (0x452556)#012/usr/local/go/src/pkg/runtime/panic.c:482 (0x452dfd)#012/usr/local/go/src/pkg/runtime/panic.c:433 (0x452c17)#012/home/vagrant/src/packetbeat/main.go:123 (0x425dcd)#012/home/vagrant/src/packetbeat/main.go:339 (0x4274f1)#012/usr/local/go/src/pkg/runtime/proc.c:220 (0x45432f)#012/usr/local/go/src/pkg/runtime/proc.c:1394 (0x456860)

No idea where it's getting 'home/vagrant' from as we don't use vagrant.

Config:

[elasticsearch]
host  = "int-elastic-01"
port  = 9200
index = "packetbeat"

[interfaces]
device = "any"

[agent]
name= "int-nginx-02"

[runoptions]
uid=450
gid=450

[procs]

  [procs.monitored.nginx]
  cmdline_grep = "nginx"

Refresh pids doesn't work after the first run

The effect is that we fail to notice when a process forks new processes or kills old ones.

[http.go] Add new field (user) when parsing HTTP request

Enhancement

Hi there! First of all, thanks for this easy-to-use soft! When using Kerberos auth, HTTP request contains a specific field which looks like [email protected] in order to transport user accessing webapp. Will it be possible to parse (and push to ElasticSearch) this field in order to make report based on it ?

Access log example

192.168.1.100 - [email protected] [25/Jun/2014:13:49:04 +0200] "GET /myapp/ HTTP/1.1" 200 10419 "-" "Mozilla/5.0 (Windows NT 6.1; rv:30.0) Gecko/20100101 Firefox/30.0"

Thanks.

virtualization guest support?

host server running SuSE Enterprise Linux 11 SP2 with XEN kernel running.

Guest OS is also SuSE Enterprise Linux 11 SP2 + application like Apache/MySQL

Does Packetbeat support the above guest traffic capturing and what is the recommended deployment of packetbeat agent ? with the application or running standalone in a dedicated vm guest OS?

Thanks,

change the elasticsearch index

Is it possible to change the elasticsearch index without rebuilding package?

Support elasticsearch not running on /

It would be nice if you could specify the context path to your elasticsearch server since we are running our on:

http://es.example.com:80/elasticsearch"

mysql.go:411: Response from unknown transaction

Is there any debug mode which will allow me to understand what's going on with this faulty transaction ?

Jun 25 14:42:37 SERVER /usr/bin/packetbeat[3698]: mysql.go:411: Response from unknown transaction. Ignoring.
Jun 25 14:42:57 SERVER /usr/bin/packetbeat[3698]: mysql.go:411: Response from unknown transaction. Ignoring.
Jun 25 14:43:17 SERVER /usr/bin/packetbeat[3698]: mysql.go:411: Response from unknown transaction. Ignoring.
[...]

Thanks,

pgsql.go:275: WARN Postgresql Message too short. 53 (length=1). Wait for more.

I have postgresql running and a packetbeat agent.
packetbeat -e -c /etc/packetbeat/packetbeat.conf -d "pgsql"
give me this output : pgsql.go:275: WARN Postgresql Message too short. 53 (length=1). Wait for more.
Do i have to do something posgresql configuration ?

netstat -lnp : +1: Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1253/sshd       
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      18983/postgres  

tcp        0      0 10.2.0.6:16001          0.0.0.0:*               LISTEN      903/python      

tcp6       0      0 :::22                   :::*                    LISTEN      1253/sshd       

tcp6       0      0 :::5432                 :::*                    LISTEN      18983/postgres  

udp        0      0 0.0.0.0:68              0.0.0.0:*                           801/dhclient3

Add limited support for SSL monitoring

/usr/bin/packetbeat[1812]: pgsql.go:275: Postgresql Message too short. 53 (length=1). Wait for more.

Relates to #46 - except I don't want to disable SSL on our database servers!
Is there any way we can still use packetbeat to see flows with PostgreSQL and SSL?

run with nice

the config file or by default should run in a nice mode.

The "default config" fallback doesn't work properly

As figured out in #2, when there's an error in the config file, packetbeat logs that it will use the defaults. However, the following pcap.Openlive fails to open device:

# packetbeat -e -c /root/packetbeat.conf 
TOML config parsing failed on /root/packetbeat.conf: Near line 21, key 'protocols.http.ports': Near line 21: Expected an array value terminator ',' or an array terminator ']', but got '8' instead.. We will use defaults.
main.go:225: ERR  pcap.Openlive failed(): : No such device exists (SIOCGIFHWADDR: No such device))

I don't find any reference to defaults in the code either, but I've never used toml before...

Thanks
Stephan

Add support for postgresql database

Packetbeat RPM config file is set to be executable

The /etc/packetbeat/packetbeat.conf file should not have mod +x

$ rpm -ql -v packetbeat
-rwxr-xr-x    1 root    root                     2920 Jun 19 00:56 /etc/packetbeat/packetbeat.conf
-rwxr-xr-x    1 root    root                     2288 Jun 19 00:56 /etc/rc.d/init.d/packetbeat
-rwxr-xr-x    1 root    root                  6113216 Jun 19 00:56 /usr/bin/packetbeat

packetbeat 32 bit error while installing

while installing packetbeat its shows error on libpcap.so.1 needed
but libpcap.so.1 is in /usr/lib
how can i fix it
pls find error shows below

�]0;root@localhost:/opt�[root@localhost opt]# rpm -ivh pak���[Kcketbeat-0.2.1-1.el6.i686.rpm
error: Failed dependencies:
libpcap.so.1 is needed by packetbeat-0.2.1-1.el6.i686
rpmlib(FileDigests) <= 4.6.0-1 is needed by packetbeat-0.2.1-1.el6.i686
rpmlib(PayloadIsXz) <= 5.2-1 is needed by packetbeat-0.2.1-1.el6.i686
�]0;root@localhost:/opt�[root@localhost opt]#
�]0;root@localhost:/opt�[root@localhost opt]# cd /usr/lib�
�]0;root@localhost:/usr/lib�[root@localhost lib]#
�]0;root@localhost:/usr/lib�[root@localhost lib]# ls -l libpcap*
�
lrwxrwxrwx 1 root root 12 Jun 13 11:58 �[00;36mlibpcap.so�[00m -> �[00;32mlibpcap.so.1�[00m
lrwxrwxrwx 1 root root 16 Jun 22 2012 �[00;36mlibpcap.so.0�[00m -> �[00;32mlibpcap.so.0.9.4�[00m
lrwxrwxrwx 1 root root 16 Jun 22 2012 �[00;36mlibpcap.so.0.9�[00m -> �[00;32mlibpcap.so.0.9.4�[00m
-rwxr-xr-x 1 root root 167096 Jan 27 2010 �[00;32mlibpcap.so.0.9.4�[00m
lrwxrwxrwx 1 root root 16 Jun 13 11:58 �[00;36mlibpcap.so.1�[00m -> �[00;32mlibpcap.so.1.1.1�[00m
-rwxr-xr-x 1 root root 233209 Jun 13 11:58 �[00;32mlibpcap.so.1.1.1

[feature] DNS

Would be nice to have DNS traffic support. Maybe adding a modular approach could speed up contribution

my log files are flooded with a message regarding redis.go

I have this message in my log file:

usr/bin/packetbeat[26848]: redis.go:482: Response from unknown transaction. Ignoring.

It's just a warning, so I think more help is welcome for log configuration in packetbeat.conf.

For now I comment redis lines in protocols section because right know I don't have transaction.

packetbeat agent get so much of Wrong length messages and cpu usage is high

Sep 3 15:23:04 redis67 /usr/bin/packetbeat[14881]: redis.go:295: Wrong length of data: 1376 instead of 1480
Sep 3 15:23:04 redis67 /usr/bin/packetbeat[14881]: redis.go:295: Wrong length of data: 1376 instead of 1536
Sep 3 15:23:04 redis67 /usr/bin/packetbeat[14881]: redis.go:254: Failed to read number of bulk messages: strconv.ParseInt: parsing "2_4": invalid syntax
Sep 3 15:23:04 redis67 /usr/bin/packetbeat[14881]: redis.go:295: Wrong length of data: 46 instead of 43
Sep 3 15:23:04 redis67 /usr/bin/packetbeat[14881]: redis.go:295: Wrong length of data: 1376 instead of 1582
Sep 3 15:23:04 redis67 /usr/bin/packetbeat[14881]: redis.go:295: Wrong length of data: 1376 instead of
Sep 3 15:23:05 redis67 /usr/bin/packetbeat[14881]: redis.go:281: Failed to read bulk message: strconv.ParseInt: parsing "39_24": invalid syntax

the protocol parser error(response message is wrong)

redis.request INCR goshop:tracking:ticket:step_id
redis.response HSETNX goshop:tracking:ticket:995585455:info ctime 1409729334
request_raw INCR goshop:tracking:ticket:step_id
response_raw HSETNX goshop:tracking:ticket:995585455:info ctime 1409729334

cpu is high (top output)

14881 root 20 0 233m 12m 7460 S 103.7 0.0 2:28.34 packetbeat

Packetbeat binary always going with localhost:9200

Even though I have updated the packetbeat.conf file to point to different server, I think packetbeat is trying to connect to local address.

Error:
Publish failure: Post http://localhost:9200/packetbeat-2014.06.12/http??: dial tcp 127.0.0.1:9200: connection refused

Using the 64 bit RPM on a centos machine

Random Crashes

The process crashes on its own and have to restart it.

Demo Site is down

Demo Site demo.packetbeat.com is 404

Alternate protocols

This project looks super interesting. I am curious though, do you support alternate protocols? We would be super interested in sip and RTP for example

Use Golang HTTP Server for frontend?

Why not suggest using (or even include) Go as the static http server?
https://code.google.com/p/go-wiki/wiki/HttpStaticFiles

Packetbeat RPM config file should be marked as config

By adding %config in the spec file infront of all config files. RPM will ensure that locally changed config files will not be overwritten when the package is upgrade.

Config files in a package can be listed with:

$ rpm -ql --configfiles packetbeat
$ 

See:
http://www.rpm.org/max-rpm/s1-rpm-inside-files-list-directives.html

Packet Handling

I can see from main.go that you are working with ipv4 packets and thought it might be worth highlighting this repository:

code.google.com/p/go.net/ipv4

http://godoc.org/code.google.com/p/go.net/ipv4

http responsetime maybe need add rtt?

I found that http responsetime is calculated from receivedHttpRequest to receivedHttpResponse. That means it only contain times nginx used, how can we measure the client time? Can we add RTT here?

Incorrectly trying to access localhost for elasticsearch

On server A with IP 192.168.1.2, I have my elasticsearch host set to Server B like so in /etc/packebeat/packetbeat.conf:

[elasticsearch]
# Set the host and port where to find Elasticsearch.
host = "192.168.1.1"  # Server B
port = 9200

But in my logs for Server A, I see packetbeat trying to access localhost:9200:

/usr/bin/packetbeat[3725]: redis.go:500: Publish failure: Post http://localhost:9200/packetbeat-2014.06.27/redis??: dial tcp 127.0.0.1:9200: connection refused
/usr/bin/packetbeat[3725]: publish.go:244: core.SearchRequest fails with: Get  http://localhost:9200/packetbeat-topology/server-ip/_search: dial tcp 127.0.0.1:9200: connection refused

Shouldn't it be trying 192.168.1.1:9200 as configured above in packetbeat.conf? I do see packetbeat data on Server B though. And I see this when I start packetbeat:

/usr/bin/packetbeat[6713]: publish.go:342: Using http://192.168.135.92:9200 as publisher

runtime error: slice bounds out of range.

Running the latest version of PB and libpcap on Ubuntu 12.04;

markw@na0-af-csb-00:~$ dpkg -l|egrep -e "packetbeat|libpcap"
ii  libpcap0.8                            1.1.1-10                            system interface for user-level packet capture
ii  packetbeat                            0.2.0-1                             Packetbeat Agent

Wasn't seeing anything in ES and when I checked the syslog saw a few errors. Tried running it manually with debug (I hope this is right);

markw@na0-ah-csb-00:~$ sudo /usr/bin/packetbeat -e -c /etc/packetbeat/packetbeat.conf -d "http,httpdetailed"
publish.go:316: INFO Use http://172.30.5.15:9200 as publisher
publish.go:326: INFO No agent name configured, using hostname 'na0-ah-csb-00'
procs.go:101: INFO Local IP addresses are: [127.0.0.1 172.30.5.5 ::1 fe80::a236:9fff:fe14:75e2]
log_unix.go:113: ERR  decodePktEth exception. Recovering, but please report this: runtime error: slice bounds out of range.
log_unix.go:114: ERR  Stacktrace: /home/vagrant/src/packetbeat/log_unix.go:114 (0x4080de)
/usr/local/go/src/pkg/runtime/panic.c:248 (0x434f46)
/usr/local/go/src/pkg/runtime/panic.c:482 (0x4357ed)
/usr/local/go/src/pkg/runtime/panic.c:439 (0x435637)
/home/vagrant/src/packetbeat/main.go:156 (0x408d79)
/home/vagrant/src/packetbeat/main.go:318 (0x40a4aa)
/usr/local/go/src/pkg/runtime/proc.c:220 (0x436d1f)
/usr/local/go/src/pkg/runtime/proc.c:1394 (0x439250)

I am monitoring a few custom ports and processes;

  [protocols.cassandra]
  ports = [7199,9160]

  [procs.monitored.java]
  cmdline_grep = "java"

Let me know if there is more I can provide!

[question] Failed to publish topology

I followed the quick start guide and got this error:

Jul 5 09:12:39 vm00 /usr/bin/packetbeat[7866]: publish.go:342: Using http://localhost:9200/opt/elasticsearch as publisher
Jul 5 09:12:39 vm00 /usr/bin/packetbeat[7866]: publish.go:343: Using index pattern [packetbeat-]YYYY.MM.DD
Jul 5 09:12:39 vm00 /usr/bin/packetbeat[7866]: publish.go:353: No agent name configured, using hostname 'vm00.linux.com.py'
Jul 5 09:12:39 vm00 /usr/bin/packetbeat[7866]: publish.go:371: Failed to publish topology: Put http://localhost:9200/opt/elasticsearch/packetbeat-topology/server-ip/10.1.1.108: malformed HTTP status code "handler"
Jul 5 09:12:39 vm00 /usr/bin/packetbeat[7866]: main.go:252: Put http://localhost:9200/opt/elasticsearch/packetbeat-topology/server-ip/10.1.1.108: malformed HTTP status code "handler"
Jul 5 09:24:51 vm00 /usr/bin/packetbeat[7956]: publish.go:342: Using http://localhost:9200/opt/elasticsearch as publisher
Jul 5 09:24:51 vm00 /usr/bin/packetbeat[7956]: publish.go:343: Using index pattern [packetbeat-]YYYY.MM.DD
Jul 5 09:24:51 vm00 /usr/bin/packetbeat[7956]: publish.go:353: No agent name configured, using hostname 'vm00.linux.com.py'
Jul 5 09:24:51 vm00 /usr/bin/packetbeat[7956]: publish.go:371: Failed to publish topology: Put http://localhost:9200/opt/elasticsearch/packetbeat-topology/server-ip/10.1.1.108: malformed HTTP status code "handler"
Jul 5 09:24:51 vm00 /usr/bin/packetbeat[7956]: main.go:252: Put http://localhost:9200/opt/elasticsearch/packetbeat-topology/server-ip/10.1.1.108: malformed HTTP status code "handler"

The only difference is that I installed ElasticSearch 1.2.1, could that be the cause?

[question][feature] Unique repository for analysis apps

Hi, would it be possible to use the same captured repository for Packetbeat, Moloch, ntop-ng and Wireshark for example?.

As I understand (still couldn't finish my Packetbeat lab), Packetbeat only will show you traffic about applications it understands, would be nice to see that with PB. In case other tools like Moloch are need, would be nice to use the same backend repository instead of having 3 copies of the same data.

daemonize not available for EL 7

packetbeat rpm depends on the daemonize package which is not available in CentOS 7/RHEL 7 and EPEL 7 beta. Hence packet beat don't install on EL 7.

Misisng Responses and transactions

I am getting a lot of the following.

http.go:495: WARN Response from unknown transaction. Ignoring.
http.go:434: WARN Two requests without a response. Dropping old request

MySQL method not available when SELECT is prefixed with new line

Hi,
Some of our code includes statements formatted like below:

"""
SELECT *
FROM
"""

Which includes new line in front of select.
Such query is not correctly recognized and mysql method is not set.

P.S. Love the project goals !

Fails to start with pcap.OpenLive failed(): : No such device exists

Hi,

trying out packetbeat, I noticed that it dies on all of our servers with this:

May  8 14:32:42 node21 2014-05-08T14:32:42Z node21 /usr/bin/packetbeat[17772]: main.go:225: pcap.Openlive failed(): : No such device exists (SIOCGIFHWADDR: No such device)

It device is set to any and besides some ports nothing was changed in packetbeat.conf. Any hints would be much appreciated, and I'd gladly provide more information...

The environment in question:

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.4 LTS"
$ uname -a
Linux node21 3.11.0-15-generic #25~precise1-Ubuntu SMP Thu Jan 30 17:39:31 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
$ aptitude show libpcap0.8
Package: libpcap0.8
State: installed
Automatically installed: no
Multi-Arch: same
Version: 1.1.1-10
#...

Thanks a lot
Stephan

FAQ or something for debug of users ? (I get issue with mysql and no place for asking or getting FAQ)

Hi,

I'm very excited with this project.
I get it working in few minutes, unless the mysql part.
Nothing inside the dashboard, no data (widgets are ok)

I have been checking my conf, and change to listen any adress instead of localhost, but same.
I'm using MariaDB, don't know if it's a problem.

That's really easy and quick to use, but when it's not working, there is not much doc.

Thanks

Cédric

[feature] Oracle

Would be nice to have Oracle RDBMS traffic support. Maybe adding a modular approach could speed up contribution

panic: runtime error: slice bounds out of range

# packetbeat -e -c /etc/packetbeat/packetbeat.conf
....
mysql.go:357: WARN Two requests without a Response. Dropping old request
mysql.go:412: WARN Response from unknown transaction. Ignoring.
mysql.go:412: WARN Response from unknown transaction. Ignoring.
mysql.go:412: WARN Response from unknown transaction. Ignoring.
mysql.go:357: WARN Two requests without a Response. Dropping old request
mysql.go:357: WARN Two requests without a Response. Dropping old request
mysql.go:357: WARN Two requests without a Response. Dropping old request
mysql.go:357: WARN Two requests without a Response. Dropping old request
mysql.go:357: WARN Two requests without a Response. Dropping old request
mysql.go:357: WARN Two requests without a Response. Dropping old request
mysql.go:357: WARN Two requests without a Response. Dropping old request
mysql.go:357: WARN Two requests without a Response. Dropping old request
panic: runtime error: slice bounds out of range

goroutine 1 [syscall]:

goroutine 2 [syscall]:

goroutine 8 [chan receive]:
main.(*PublisherType).UpdateTopologyPeriodically(0xa19a00)
        /home/vagrant/src/packetbeat/publish.go:178 +0x3c
created by main.(*PublisherType).Init
        /home/vagrant/src/packetbeat/publish.go:319 +0x511

goroutine 6 [IO wait]:
net.runtime_pollWait(0x7fc541104f00, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2000bb350, 0xb, 0xc2000d52d0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc2000bb2d0, 0xc2001e4000, 0x1000, 0x1000, 0x0, ...)
        /usr/local/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc2001d9ea0, 0xc2001e4000, 0x1000, 0x1000, 0x0, ...)
        /usr/local/go/src/pkg/net/net.go:123 +0xc3
bufio.(*Reader).fill(0xc2000c57e0)
        /usr/local/go/src/pkg/bufio/bufio.go:79 +0x10c
bufio.(*Reader).Peek(0xc2000c57e0, 0x1, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/bufio/bufio.go:107 +0xc9
net/http.(*persistConn).readLoop(0xc2001c4d80)
        /usr/local/go/src/pkg/net/http/transport.go:670 +0xc4
created by net/http.(*Transport).dialConn
        /usr/local/go/src/pkg/net/http/transport.go:511 +0x574

goroutine 7 [select]:
net/http.(*persistConn).writeLoop(0xc2001c4d80)
        /usr/local/go/src/pkg/net/http/transport.go:774 +0x26f
created by net/http.(*Transport).dialConn
        /usr/local/go/src/pkg/net/http/transport.go:512 +0x58b

goroutine 9 [chan send (nil chan)]:
main.(*Process).RefreshPids(0xc200165c80)
        /home/vagrant/src/packetbeat/procs.go:139 +0x399
created by main.NewProcess
        /home/vagrant/src/packetbeat/procs.go:123 +0x8e

goroutine 10 [chan send (nil chan)]:
main.(*Process).RefreshPids(0xc200165cd0)
        /home/vagrant/src/packetbeat/procs.go:139 +0x399
created by main.NewProcess
        /home/vagrant/src/packetbeat/procs.go:123 +0x8e

goroutine 11 [chan send (nil chan)]:
main.(*Process).RefreshPids(0xc200165d20)
        /home/vagrant/src/packetbeat/procs.go:139 +0x399
created by main.NewProcess
        /home/vagrant/src/packetbeat/procs.go:123 +0x8e

goroutine 12 [finalizer wait]:

goroutine 98 [IO wait]:
net.runtime_pollWait(0x7fc541104dc0, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2002071a0, 0xb, 0xc2000d52d0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc200207120, 0xc200483000, 0x1000, 0x1000, 0x0, ...)
        /usr/local/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc200666088, 0xc200483000, 0x1000, 0x1000, 0x0, ...)
        /usr/local/go/src/pkg/net/net.go:123 +0xc3
bufio.(*Reader).fill(0xc200218180)
        /usr/local/go/src/pkg/bufio/bufio.go:79 +0x10c
bufio.(*Reader).Peek(0xc200218180, 0x1, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/bufio/bufio.go:107 +0xc9
net/http.(*persistConn).readLoop(0xc2001b3c80)
        /usr/local/go/src/pkg/net/http/transport.go:670 +0xc4
created by net/http.(*Transport).dialConn
        /usr/local/go/src/pkg/net/http/transport.go:511 +0x574

goroutine 99 [select]:
net/http.(*persistConn).writeLoop(0xc2001b3c80)
        /usr/local/go/src/pkg/net/http/transport.go:774 +0x26f
created by net/http.(*Transport).dialConn
        /usr/local/go/src/pkg/net/http/transport.go:512 +0x58b
#

I haven't captured the packets that created this (as I have no clue what exactly caused this), but maybe the traceback already helps...?

Cheers
Stephan

Unable to get mysql data

Hey.
I'm trying to use packetbeat to monitor my LAMP stack activity running on my ubuntu 12.04.4
Unfortunately, I cannot see the mysql data come into the kibana dashboard but I can monitor my http trafic.

First my problem was coming from using sockets instead of the network. But now, even if I can see the mysql network trafic into wireshark/packetbeat -e/tcpdump on 3306 port, it never ends displaying into my kibana dashboard.

While running the "sudo packetbeat -e -c /etc/packetbeat/packetbeat.conf -d mysql,mysqldetailed" command, I can see lines like these :

mysql.go:115: DBG MySQL Header: Packet length 91, Seq 0, Type=10 mysql.go:115: DBG MySQL Header: Packet length 92, Seq 1, Type=141

That can be followed by others like these (not occuring all the time):

procs.go:253: ERR FindSocketsOfPid: Open: open /proc/4945/fd: no such file or directory procs.go:253: ERR FindSocketsOfPid: Open: open /proc/13855/fd: no such file or directory

Do you have any idea how I may solve this ?

Thank you.

RPM fails on preun script

The preun script in the RPM fails on %doc

$ sudo rpm -e -vv packetbeat
[...]
D: opening  db index       /var/lib/rpm/Triggername create mode=0x42
D:     erase: %preun(packetbeat-0.2.1-1.el6.x86_64) scriptlet start
D:     erase: %preun(packetbeat-0.2.1-1.el6.x86_64)     execv(/bin/sh) pid 25877
+ /etc/init.d/packetbeat stop
Stopping packetbeat agent: [FAILED]
+ /sbin/chkconfig --del packetbeat
+ %doc
/var/tmp/rpm-tmp.aJ8vAC: line 5: fg: no job control
D:     erase: waitpid(25877) rc 25877 status 100 secs 0.018
error: %preun(packetbeat-0.2.1-1.el6.x86_64) scriptlet failed, exit status 1
D: running post-transaction scripts
D: closed   db index       /var/lib/rpm/Triggername
D: closed   db index       /var/lib/rpm/Basenames
D: closed   db index       /var/lib/rpm/Name
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm

integration with docker and CoreOS

Not sure what the best approach is here. One could attempt to run packetbeat on the host that runs docker images. But on CoreOS the OS is read-only and everything is supposed to be ran from a container. So it might make more sense to run this inside a docker container.

Running Packetbeat and Logstash on the same Elasticsearch instance

Is it possible to update documentation for people who already have Elasticsearch and Kibana installed.

In this case you have just to install packetbeat and import dashboards. You may think this is logic but in my opinion, you should be interesting for newbies and so on.

In documentation there is "We have extended Kibana to support new panels specialised in visualising network data. So it is best to download it from packetbeat GitHub account: " but I make a diff between my Kibana version and this version so it look like the same except config.js file.

Thanks,
Charles

HTTPS traffic...

Hi Guys and thanks for this amazing tool.
I was wondering if monitoring HTTPS traffic may be possible. I configured my clients to sniff on port 443 but so far I can't see any output over Kibana, only HTTP transactions are displayed. Or did I forget to configure something?
Though I think it would be a great improvement for Packetbeat.
Cheers

Add log output

It would be nice to be able to configure the output to just go to a log file for it's output instead of Elasticsearch directly.
As the number of systems running the agent grows, things like compression and bulk indexing become necessary.
Those specific features "could" be coded into this application but outputting to a log would allow for other more generalized transport mechanisms to pick up the data.
i.e. logstash, heka, fluentd.

No client IPs

Hi, I did a mini POC, and was only able to see the two servers I added in the node graph. Shouldn't I be seeing the client hosts also?

Can't start packet beat when agent name is set

With 1.1-1 on CentOS (rpm downloaded from site)

When I set the name pareter in the agent section I can't get packetbeat started.
When I don't set the name it works perfectly.

Negative responsetime for postgresql

Negative responsetime for postgresql transactions.

apt repo

It would be great if packetbeat had a proper apt repo, this would make keeping on top of releases and automating installs a lot easier