AWS CloudWatch logs is an useful logging system, but it has two quircks. It does not allow you too set a default retention period for newly created log groups, and it does not delete empty log streams that are older than the retention period. This utility:
- sets a default retention period on log groups without a period set.
- removes empty log streams older than the retention period of the log group
You can use it as a command line utility. You can also install it as an AWS Lambda function and have your logs kept in order, NoOps style!
to install the log minder, type:
pip install aws-cloudwatch-log-minder
to set the default retention period on log groups without one, type:
cwlog-minder --dry-run set-log-retention --days 30
This will show you which log groups will have its retention period set. Remove the --dry-run
and
it the retention period will be set.
To delete empty log streams older than the retention period, type:
cwlog-minder --dry-run delete-empty-log-streams
This will show you which empty log streams will be deleted. Remove the --dry-run
and
these stream will be deleted.
To deploy the log minder as an AWS Lambda, type:
git clone https://github.com/binxio/aws-cloudwatch-log-minder.git
cd aws-cloudwatch-log-minder
aws cloudformation deploy \
--capabilities CAPABILITY_IAM \
--stack-name aws-cloudwatch-log-minder \
--template-file ./cloudformation/aws-cloudwatch-log-minder.yaml \
--parameter-overrides LogRetentionInDays=30
This will install the log minder in your AWS account and run every hour.
export LOG_LEVEL=INFO
cwlog-minder ...
AWS regions and credential profiles can be selected via command line arguments or environment variables.
cwlog-minder --region eu-west-1 ...
export AWS_DEFAULT_REGION=eu-west-1
cwlog-minder ...
cwlog-minder --profile dev ...
export AWS_PROFILE=dev
cwlog-minder ...