aramalipoor / aws-cost-saver

A tiny CLI tool to help save costs in development environments when you're asleep and don't need them!

aws-cost-saver

A tiny CLI tool to help save costs in development environments when you're asleep and don't need them!

• Usage
  • Commands
    • conserve
    • restore
  • State-file
    • AWS S3
    • Local File System
• Tricks
  1. shutdown-ec2-instances
  2. stop-fargate-ecs-services
  3. stop-rds-database-instances
  4. decrease-dynamodb-provisioned-rcu-wcu
  5. remove-nat-gateways
  6. snapshot-and-remove-elasticache-clusters
  7. decrease-kinesis-streams-shards
  8. stop-rds-database-clusters
  9. scaledown-auto-scaling-groups
  10. suspend-auto-scaling-groups

Disclaimer

This utility is meant for development environments only where stopping and removing resources is not risky.

Usage

# Install
$ npm install -g aws-cost-saver

# Try
$ aws-cost-saver conserve --help
$ aws-cost-saver conserve --dry-run --no-state-file
$ aws-cost-saver conserve --dry-run --no-state-file --only-summary
$ aws-cost-saver conserve -d -n --tag Team=Tacos
$ aws-cost-saver conserve -d -n -t Team=Tacos -t Application=Orders

# Use
$ aws-cost-saver conserve
$ aws-cost-saver restore

Commands

Under the hood aws-sdk is used, therefore AWS Credentials are read in this order:

1. From AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_REGION environment variables.
2. From shared ini file (i.e. ~/.aws/credentials)

Conserve

This command uses various tricks to conserve as much money as possible. To be able to restore, this command will create a state-file.

USAGE
  $ aws-cost-saver conserve [-d|--dry-run] [-s|--state-file aws-cost-saver.json] ...

OPTIONS
  -d, --dry-run                          Only print actions and write state-file of current resources.
  -n, --no-state-file                    Ignore saving current state, useful when want to only conserve as much money as possible.
  -s, --state-file state-file.json       [default: file://aws-cost-saver.json] Where to keep original state of stopped/decreased resources to restore later.
  -w, --overwrite-state-file             Overwrite state-file if it exists. WARNING: Use with caution as this might overwrite non-restored state-file.
  
  -u, --use-trick trick-machine-name     Enables an individual trick. Useful for tricks that are disabled by default. Can be used multiple times.
  -i, --ignore-trick trick-machine-name  Disables an individual trick. Useful when you do not like to use a specific trick. Can be used multiple times.
  --no-default-tricks                    Disables all default tricks. Useful alongside --use-trick to enable only specific set of tricks.
  
  -t, --tag Name[=Value]                 Tags to narrow down targeted resources. Multiple tags will be AND-ed. Providing "Value" is optional.
  -r, --region eu-central-1              [default: eu-central-1] AWS Region to converse resources in.
  -p, --profile my-aws-profile           [default: default] AWS Profile to use from ~/.aws/config
  -m, --only-summary                     Do not render live progress. Only print final summary in a clean format.
  -h, --help                             Show CLI help

Restore

To restore AWS resources stopped or removed by the conserve command.

USAGE
  $ aws-cost-saver restore [-d|--dry-run] [-s|--state-file aws-cost-saver.json] ...

OPTIONS
  -d, --dry-run          Only list actions and do not actually execute them.
  -s, --state-file       [default: file://aws-cost-saver.json] Path to load previous state of your AWS resources from.
  
  -r, --region           [default: eu-central-1] AWS region to restore resoruces in.
  -p, --profile          [default: default] AWS profile to lookup from ~/.aws/config
  -m, --only-summary     Do not render live progress. Only print final summary in a clean format.
  -h, --help             Show CLI help.

State-file

When resources are conserved their current state (e.g. number of shards, number of instances, etc.) will be saved in a state-file to be to used to restore at a later time.

At the moment when you restore the resources it's up to you to either remove the state-file or save it for a later use.

Supported storage providers

aws-cost-saver supports file: and s3: storage providers.

AWS S3

$ aws-cost-saver conserve --dry-run --state-file s3://my-bucket-name/some-dir/aws-cost-saver.json

Local File System

$ aws-cost-saver conserve --dry-run --state-file file://./aws-cost-saver.json
$ aws-cost-saver conserve --dry-run --state-file file:///etc/aws-cost-saver.json

# Local file system is the default storage
$ aws-cost-saver conserve --dry-run --state-file ./aws-cost-saver.json
$ aws-cost-saver conserve --dry-run --state-file /etc/aws-cost-saver.json

Keeping an ideal state-file

One use-case is to keep an ideal state in a state-file (e.g. result of first time you run conserve) then always conserve resources without keeping the state and restore from the ideal state-file.

For example on day 1:

# Generate a state-file of current resources:
aws-cost-saver conserve --dry-run --state-file ideal-state.json

# Manually update numbers if you like:
vi ideal-state.json

Then the daily routine can look like this:

# In the morning, bring back the ideal state:
aws-cost-saver restore --state-file ideal-state.json

# ... develop amazing software :D

# At night, conserve as much as possible without keeping the state:
aws-cost-saver conserve --no-state-file

Tricks

Here is a list of tricks aws-cost-saver uses to reduce AWS costs when you don't need them.

# shutdown-ec2-instances

Stopping running EC2 instances will save compute-hour. This trick will keep track of stopped EC2 instances in the state-file and start them again on restore.

# stop-fargate-ecs-services

Stopping AWS Fargate ECS services (i.e. tasks) will save compute-hour. This trick will keep track of stopped Fargate ECS services in the state-file and start them again on restore.

# stop-rds-database-instances

Stopping RDS databases will save underlying EC2 instance costs. This trick will keep track of stopped databases in the state-file and start them again on restore.

# decrease-dynamodb-provisioned-rcu-wcu

Provisioned RCU and WCU on DynamoDB tables costs hourly. This trick will decrease them to minimum value (i.e. 1). Original values will be stored in state-file to be restored later.

# remove-nat-gateways

NAT Gateways are charged hourly. This trick will remove NAT Gateways while you don't use your services, and creates them again on "restore" command.

• Removing NAT Gateways stops instances access to internet.
• This trick is currently disabled by default because removing/recreating NAT gateway will change the ID therefore IaC such as terraform will be confused. Use --use-trick flag to explicitly enable it:

$ aws-cost-saver conserve --use-trick remove-nat-gateways

# snapshot-and-remove-elasticache-clusters

ElastiCache clusters cost hourly but unfortunately it's not possible to stop them like an EC2 instance. To save costs this trick will take a snapshot of current cluster (preserving data, config and cluster ID) and delete it. To restore it'll create a new cluster based on snapshot taken.

• Due to AWS limitation, backup and restore is supported only for clusters running on Redis.
• This trick is currently disabled by default to be tested by early users. Use --use-trick flag to explicitly enable it:

$ aws-cost-saver conserve --use-trick snapshot-remove-elasticache-redis

# decrease-kinesis-streams-shards

Kinesis Stream Shards cost hourly. This trick will decrease open shards to the minimum of 1, in multiple steps by halving number of shards in each step. Currently this trick is useful when you're doing UNIFORM_SCALING, i.e. default config of Kinesis Stream.

# stop-rds-database-clusters

Stopping RDS clusters will save underlying EC2 instance costs. This trick will keep track of stopped clusters in the state-file and start them again on restore.

# scaledown-auto-scaling-groups

When Auto Scaling Groups are configured they might launch EC2 instances. This trick will set "desired", "min" and "max" capacity of ASGs to zero and keep track of original values in the state-file. Scaling-down an ASG will terminate all instances therefore temporary volumes will be lost.

• This trick is currently disabled by default. Use --use-trick flag to explicitly enable it:

$ aws-cost-saver conserve --use-trick scaledown-auto-scaling-groups

# suspend-auto-scaling-groups

When Auto Scaling Groups processes are active they might launch EC2 instances. This trick will suspend all processes of ASGs to prevent launching new instances.

# etc

If you know any other tricks to save some money feel free to create a Pull Request or raise an issue.

Alternatives

There are various ways to save money on AWS that need per-case judgement and it'll be hard to generalize into aws-cost-saver, but here is a list of useful resources:

• Google Search: "aws cost saving"
• Use Amazon EC2 Spot Instances to reduce EC2 costs for background and non-critical services
• Delete idle LBs, Use private subnets, Use auto-scalers, etc.

License

AWS Cost Saver is licensed under MIT License. See LICENSE for the full license text.

aws-cost-saver's Issues

trick: Decrease shards of a Kinesis Stream

Suggested by weeefun on HN

.

.

Issue: Error on stop-fargate-ecs-services

Version - [email protected]
Service/Trick - stop-fargate-ecs-services

I have around 23 ECS tasks running. The current latest version is throwing this error though the stop-fargate-ecs-services is successful and in AWS I could see all tasks desired count is set to 0.
I guess this is because of some timeout settings.

Error -
✔ partially finished, with 1 failed tricks out of 3.

◎ conserve: stop-fargate-ecs-services
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
  ✔ service/*******
   ✔ desired count
   set desired count to zero
   ↓ auto scaling
    ↓ no scalable targets defined
************************
✔ partially finished, with 1 failed tricks out of 3.
    Error: ConservePartialFailure

Add support to aurora-postgresql DB instances

aurora-PostgreSQL DB instances are not eligible for stopping and starting.

Issue: OVERWRITE existing state-file

When using --state-file s3:// on a daily scheduled cron., the job fails for the second time and further with error

→ State file already exists: s3://***/staging-state-file.json ? Are you sure you want to OVERWRITE existing state-file? You will not be able to restore to previously conserved resources! (y/N)

Please enable overwrite flag while using --state-file s3://

Add Support for environment variables to configure credentials

Hello,
I'd like to suggest adding support for environment variables for profile, access key id, secret access key, default region, etc. replicating what the AWS cli and SDK's support.

I'm happy to contribute as well but wanted to see your interest first. :-)

Suspend/Enable Autoscaling Groups

Autoscaling groups kick in once you shut down the EC2 instances. To prevent this you must suspend them.
It would be great if you could include them in your process.

Here are the scripts I use pre conserve and restore

Suspend Auto-scaling Groups: This will use the region from your ~/.aws/config profile

aws autoscaling describe-auto-scaling-groups --output=yaml | grep AutoScalingGroupName | awk '{print $2}' | xargs -n1 aws autoscaling suspend-processes --auto-scaling-group-name

Resume Auto-scaling Groups: This will use the region from your ~/.aws/config profile

aws autoscaling describe-auto-scaling-groups --output=yaml | grep AutoScalingGroupName | awk '{print $2}' | xargs -n1 aws autoscaling resume-processes --auto-scaling-group-name

Any commands for stopping Elasticache?

I use AWS Elasticache for Redis and also can shut that down or stop it to save some costs, really thanks for this project, looking for something like it for a while now!

feature: Allow storing state-file in S3

This will enable using aws-cost-saver in a Lambda potentially.

trick: Decrease provisioned read/write capacity of a DynamoDB table

Suggested by weeefun on HN

feature: Allow filtering by tag

Discussed by rxDotIo and twratl on reddit.

trick: Shutdown Kubernetes clusters

Suggested by adamlklein on HN