Comments (10)
Fair point, and open to suggestions: any recommendations as to what options would be decent?
from elk-docker.
Maybe something like this:
5 files with 100MB limit per service totally 500MB per service and totally over kibana, logstash, elasticsearch it would be 1500MB.
from elk-docker.
Hmm. How logstash handle log writing? Is there log4j etc. handler to handle logrotating? Few examples just point out normal daily logrorate (which should be then hourly in this case?).
from elk-docker.
OK thanks for the input, will look into this and update the image.
For Logstash at the very least, the logrotate configuration for Logstash that is included in the DEB/RPM/etc. packages looks promising.
from elk-docker.
@spujadas How do you manage the logs that is sent from the client servers to ELK servers? How can we retain only 3 days worth of data in ELK servers that gets displayed in Kibana?
from elk-docker.
@gvenka008c Sorry, not a Logstash expert so haven't got a definite answer to that one.
Deleting old indices as described here seems like a good idea, but don't take my word for it: you may want to check in with the Logstash community over at https://discuss.elastic.co/c/elasticsearch for a more solid answer to your question.
from elk-docker.
Ended up going with a daily rotation + compression + deletion after a week, using logrotate for all three services.
Tried to configure Elasticsearch's logging.yml
but that was very fiddly and ultimately dissatisfying: Elasticsearch's use of log4j 1.x does allow for log rotation but it appears to be much less flexible than logrotate (i.e. can't seem do to what logrotate easily does). This should change if/when log4j is upgraded to 2.x (see elastic/elasticsearch#17697 for upgrade plans), in the meantime will play it safe and use logrotate (tested, appears to work as intended).
Anyway, now has a "sensible" default, which can be overriden as needed.
from elk-docker.
Thanks. Sound very good!
from elk-docker.
Instead of using logrotate on logstash you may consider removing the following from 30-output.conf
stdout { codec => rubydebug }
See here: https://discuss.elastic.co/t/logstash-stdout-large-size/24939/4
Also, this logrotate setup did not work for me. After some debugging it seems as if the cron daemon is not running within the container (see issue #60)
from elk-docker.
@reallistic Yep, sounds like a good idea, but I need to do a few tests first (see #60, will have to wait a couple of weeks for that).
from elk-docker.
Related Issues (20)
- cannot add login page to kibana HOT 1
- How to use environment variable in 30-output.conf file HOT 2
- ELK fails to start on MAC M1 HOT 8
- Setting up APM question HOT 4
- Question: user authentication for https HOT 1
- Error in Security section HOT 1
- Issues installing on TrueNAS Scale HOT 1
- Update ELK to latest version (currently 8.3.3) HOT 4
- Issues running on AWS Fargate HOT 2
- Add sample docker-compose.yml with persistance + traefik configuration HOT 1
- example using image never starts as elasticsearch doesn't start HOT 1
- Kibana refuses connection, nothing in logs HOT 2
- filebeat x509 certificate signed by unknown authority when calling api endpoint HOT 1
- Kibana enrollement token
- Update ELK to 8.9.0 HOT 1
- Kibana 8.11.1 Security Update (ESA-2023-25) HOT 1
- want to move data from this running stack to another machine (arm64-amd64) how to
- how to disable the features that require a licence.
- Please update to 8.13.2 HOT 1
- For Running on MAC M2 need need bootstrap.system_call_filter: false in elasticsearch.yml HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elk-docker.