logstash logs about elk-docker

[spujadas]

Hi,

Yes, you most definitely can, and you're looking in the right place (/var/log/logstash does contain the actual logs in logstash.log, as well as Logstash's standard output and error in logstash.stdout and logstash.err), but by default Logstash's logs are remarkably quiet.

The easiest way to have more detailed output from Logstash on an ad hoc basis (i.e. without extending the image to set up a debug configuration) is to go to bash in the running container just like you did, and then:

• Stop the running instance of Logstash:

  service logstash stop
  

• Manually start Logstash in verbose mode (very verbose, outputs a Flushing buffer at interval log item every single second) or debug mode, using either

  /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log --verbose
  

  or

  /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log --debug
  

That way, you'll have Logstash's standard output and error displayed on screen, and if you open another bash session in the running container you can view/tail -f/whatnot the logs in /var/log/logstash/logstash.log.

Hope that you'll find something in all this output that helps you troubleshoot your config.

[jchannon]

Awesome. Thanks!

On Tuesday, 1 December 2015, Sébastien Pujadas [email protected]
wrote:

Hi,

Yes, you most definitely can, and you're looking in the right place (
/var/log/logstash does contain the actual logs in logstash.log, as well
as Logstash's standard output and error in logstash.stdout and
logstash.err), but by default Logstash's logs are remarkably quiet.

The easiest way to have more detailed output from Logstash on an ad hoc
basis (i.e. without extending the image to set up a debug configuration) is
to go to bash in the running container just like you did, and then:

Stop the running instance of Logstash:

service logstash stop

Manually start Logstash in verbose mode (very verbose, outputs a Flushing
buffer at interval log item every single second) or debug mode, using
either

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log --verbose

or

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log --debug

That way, you'll have Logstash's standard output and error displayed on
screen, and if you open another bash session in the running container you
can view/tail -f/whatnot the logs in /var/log/logstash/logstash.log.

Hope that you'll find something in all this output that helps you
troubleshoot your config.

—
Reply to this email directly or view it on GitHub
#10 (comment).

[spujadas]

Cheers!
Leaving open for now if you want to make sure that this does actually sort out your issue, or if something more's needed.

[jchannon]

Worked a treat, thanks! 🎉

[spujadas]

Great to hear that, thanks!

[wesleymusgrove]

@spujadas, I'm forwarding logs from filebeat into logstash and my /var/log/logstash/logstash.log and /var/log/logstash/logstash.stdout files were extremely loud, racking up 7GB while I left it running for an hour over lunch.

To prevent the logstash.stdout file from getting too big I commented out the stdout line in 30-output.conf:

output {
  elasticsearch {
  hosts => ["localhost"]
      sniffing => true
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
  }
  #stdout { codec => rubydebug }
}

In the /etc/filebeat/filebeat.yml config, I turned on logging to a rotating file, namely /var/log/mybeat/mybeat.log, which keeps 7 files and cuts over to a new one every 10MB per the default options.

Are there any options I can change in the Dockerfile or logstash.init to cause /var/log/logstash/logstash.log to be a rotating log file like mybeat.log?

[wesleymusgrove]

As a follow-up, I read this elastic discussion that mentioned:

Logstash's package includes a logrotate configuration file that causes /var/log/logstash/logstash.log to get rotated daily with a week's worth of logs kept around.

Running docker exec -it [container_id] bash and examining /etc/logrotate.conf, I can see that it includes log rotation files from various packages located in /etc/logrotate.d. Packages like apt, dpkg, rsyslog.disabled, syslog-ng, unattended-upgrades, and upstart all have logrotate config files, but there was not one listed for logstash like the quote above mentioned. So I created my own like this located at /etc/logrotate.d/logstash:

/var/log/logstash/*.log {
        maxsize 10M
        hourly
        rotate 7
        copytruncate
        compress
        delaycompress
        missingok
        notifempty
}

This will create a rolling log file every hour or whenever it hits 10M, whichever comes first, keeping the last 7 files.

The logrotate script is being called by default on a daily basis in /etc/cron.daily/logrotate, but based on this stackoverflow answer, I needed to be calling logrotate much more frequently for such large log files. So I edited crontab -e and added the logrotate job on a 5 minute schedule:

*/5 * * * * /etc/cron.daily/logrotate

Now here's my actual question...

Can you help me set up /var/log/logstash/logstash.log to be a rolling file in the Dockerfile or logstash.init because the changes I just made above do not persist container restarts.

Thanks!!

[spujadas]

@wesleymusgrove I'll outline what you need to do and include some pointers, and the documentation for Docker should fill in the gaps nicely if you need more information.

The quick and dirty way to persist your changes is to use docker commit once you've made your changes.

A better way to go about persisting the changes is to extend the base sebp/elk image (see the documentation on Extending the image for a quick overview). In your new Dockerfile add the appropriate ADD and RUN directives that replicate what you did when you docker exec'd into the container (e.g. ADD a logstash config file for logrotate in /etc/logstash.d/logstash, overwrite the 30-output.conf by ADDing your own, RUN some chmod/chown as required, ADD a crontab file entry, then RUN crontab on it…). Then run docker build to process the Dockerfile and obtain the extended image.

In addition, if you want your Logstash log files to persist across restarts, you'll also want to use a VOLUME directive (i.e. VOLUME /var/log/logstash).

Hope that helps.

[wesleymusgrove]

Thanks @spujadas! I had some trouble getting the cron daemon to run as the container was starting, but eventually got it working. Here's what I did:

I created a logstash-logrotate file as mentioned above in #10 (comment) and made the following changes to the Dockerfile:

# log rotate
ADD ./logstash-logrotate /etc/logrotate.d/logstash

# cron
COPY ./crontab /tmp/
RUN cat /tmp/crontab >> /etc/crontab \
 && rm -f /tmp/crontab

Regarding ./crontab, I got this method of appending to an existing file from moby/moby#12193, though there's probably a better way of doing it with sed, which I would gladly accept.

The contents of ./crontab are as follows. The date command was used for debugging purposes to verify that cron was running.

*/5 * * * * root run-parts --report /etc/cron.daily >> /tmp/logrotate.log 2>&1
#* * * * * root date >> /tmp/date.log 2>&1

In start.sh I kept trying to start the cron daemon by running variations of service cron start, /etc/init.d/cron start, start cron, but everytime I would get output like:

$ service cron start
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service cron start

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the start(8) utility, e.g. start cron

I learned that I simply needed to just execute the cron command in start.sh to get it running, but still the logrotate job was not getting executed. Referring to this ticket phusion/baseimage-docker#198 (comment), I resolved this issue by touching the /etc/crontab that my Dockerfile had appended the contents of ./crontab into.

/usr/sbin/cron
sleep 3
echo "# touched to kickstart crond" >> /etc/crontab

After that I could see that the cron jobs inside the /etc/cron.daily directory were being executed every 5 mins and that the logstash configuration file I had ADDed to /etc/logrotate.d/logstash was rotating my /var/log/logstash/logstash.log file.

[spujadas]

@wesleymusgrove Great to hear that you got it working. Thanks very much for the update and for the useful tip on dockerising the cron daemon.