Log rotation about malcolm

[mmguero]

There are a couple of settings that can be tweaked with regards to storage space utilized:

• OpenSearch index storage: during the configuration questions (./install.py --configure) you're asked Delete the oldest indices when the database exceeds a certain size?. If you answer y to this question, it will prompt you for a size (Enter index threshold (e.g., 250GB, 1TB, 60%, etc.)) and Determine oldest indices by name (instead of creation time)?. This last question, if answered y, will treat the indexes with the earliest data as oldest, where a n will treat the indexes with the most recently inserted data. If the data you're analyzing is live, then there's not a meaningful difference. Malcolm's OpenSearch indexes are organized by day, so you'd of course need to have enough disk space for at least a couple of days' worth of indexes. But beyond that, when your opensearch indexes go beyond the threshold specified, the oldest indexes will be deleted. You can also check out the documentation on index management.
• PCAP storage: Also during the ./install.py --configure questions, you'll be prompted Should Arkime delete PCAP files based on available storage (see https://arkime.com/faq#pcap-deletion)? You can review that link for more information there, but answering y to this question will allow Arkime's background processes to delete old PCAP files as the PCAP fills up disk space. However, by default the value in Arkime's config.ini of freeSpaceG=10% will reserve 10% of the total disk as free before deleting PCAPs. If you wish to change this value, then you'll need to make changes to your local arkime/etc/config.ini file (depending on how you installed Malcolm you may need to download this from GitHub or from a git cloned working repository), ensure it's being bind-mounted in your docker-compose file like this and then restart Malcolm.
• Zeek and Suricata log and artifact storage: In Malcolm's docker-compose files, there are values for how long before Zeek and Suricata logs are cleaned up . The settings for carving and preserving carved files from traffic may also be of interest.

I think that about covers it.