chaordic / tiopatinhas

An AWS Autoscaling companion that saves money by using instances from the Spot Market

tiopatinhas

Overview

tiopatinhas (tp) is a companion for AWS's Auto Scaling. It attaches itself to an Availability Group and adds resources bought in Amazon's Spot market.

As we can lose those instances at any time (due to market conditions), tiopatinhas only allows itself to provide about 50% of the total number of running instances.

Other features include:

• efficient use of resources. tiopatinhas is aware of Amazon's one-hour billing cycles and has protections against "flapping"
• crash recovery: in case there is a market crash, tiopatinhas can acquire instances on the regular OnDemand market
• fail-safe design: tiopatinhas fails "up". If the process crashes, the worse thing that can happen is extra servers being left in the cluster. Amazon's Autoscaling rules are not modified at all and can take over at any time.

Our theory of operations is that even when the system is live, each server is actually about 50% idle so that the system can responde to changes in access patterns. How much you can commit to Spot Instances is correlated to how low you keep your load during normal system operations. Needless to say, you shouldn't use Spot Instances in systems which are not fault tolerant.

Getting Started

Before starting

1. Make sure the boto python package is installed in your system. If you use debian or ubuntu you can install it by typing: 'sudo pip install boto'.
2. Make sure your AWS credentials are specified in a boto configuration file (typically ~/.boto). Instruction on how to setup this file can be found here.
3. To run the setup, you may need to install setuptools

Configuring tiopatinhas

• Copy the template conf file (tp.conf.template) to tp/tp.conf so that the script can read it and make the changes according to your needs. Tio patinhas currently supports the following properties:

Mandatory Properties

• max_price: A map that specifies the maximum bid prices for each type of EC2 instance. TP will use the prices specified in this map to bid for instances of that type in the spot market.
• max_candidates: The maximum number of instances TP will manage.
• instance_name: The prefix that will be used by TP to name managed instances.
• region: The AWS region where the AutoScaling group is located.
• placement: The AWS availability zone where TP instances will be launched.

Advanced properties

• spot_type: The instance type to bid for in the spot market (recommended: the same instance from the ASG)
• emergency_type: The instance type to buy in case of market crash (recommended: the same instance from the ASG)
• bid_threshold: Time to wait before doing another spot bid to AWS. Defaults to 300 seconds.
  • More information can be found here.
• cool_down_threshold: Time to wait before doing another scale action again. Defaults to 360 seconds.

Optional properties

• tags: A map containing custom metadata tags that must assigned to TP instances. (optional)
  • More information can be found here.
• user_data_file: An optional script or data that will be supplied to the instance on startup. If not provided, it will try to get from the Launch Configuration Group. (optional)
  • More information can be found here.
• subnet_id: The VPC's subnet id that will be used by instances TP will manage. (optional)
  • More information can be found here.
• monitoring_enabled: There are two types of Monitoring: basic and detailed. You can enable or disable the detailed monitoring by setting this field to True or False. (optional)
  • More information can be found here.

Coding with tiopatinhas

To install the latest version directly from GitHub:

$ git clone https://github.com/chaordic/tiopatinhas.git
$ python tiopatinhas/setup.py install

You may need to use sudo depending on your environment setup.

Then:

from tp import tp

t = tp.TPManager("auto scaling group name", debug=verbose)
t.run()

Executing tiopatinhas

To install the latest version directly from GitHub:

$ git clone https://github.com/chaordic/tiopatinhas.git
$ python tiopatinhas/setup.py install

You may need to use sudo depending on your environment setup.

• Once the tp/tp.conf file is ready, execute tiopatinhas with the following command:
  • python tp.py -g <AutoScalingGroupName> (this command must currently be executed from within the "tp" folder)
• You must optionally supply options "-v" for verbose mode or "-d" for daemon mode.

License

tiopatinhas is available under the MIT license.

tiopatinhas's Issues

Use custom metrics to scale up/down instances

Is it possible to use cloudwatch custom metrics (like load average) to scale up/down instances?

Can't find security groups

Hi guys!

I'm trying to use tiopatinhas on my ASG but it returns an error with security groups:

[root:~/tiopatinhas/tp #]python tp.py -g ASG-nginx-images-prod-20160316 -v
2016-04-28 23:38:28,338 ASG-nginx-images-prod-20160316 INFO Starting Tio Patinhas
2016-04-28 23:38:28,339 ASG-nginx-images-prod-20160316 DEBUG Refreshing state...
2016-04-28 23:38:30,432 ASG-nginx-images-prod-20160316 DEBUG Current candidate for target instances: 1.0
2016-04-28 23:38:30,432 ASG-nginx-images-prod-20160316 DEBUG >> guess_target(): changed target from 0 to 1.0
2016-04-28 23:38:30,432 ASG-nginx-images-prod-20160316 INFO >> refresh(): autoscale instance count changed from None to 1
2016-04-28 23:38:30,432 ASG-nginx-images-prod-20160316 DEBUG *** Current state:
2016-04-28 23:38:30,433 ASG-nginx-images-prod-20160316 DEBUG Managed by Autoscale: 1
2016-04-28 23:38:30,433 ASG-nginx-images-prod-20160316 DEBUG Managed by TP: 0
2016-04-28 23:38:30,433 ASG-nginx-images-prod-20160316 DEBUG Target: 1.0
2016-04-28 23:38:30,433 ASG-nginx-images-prod-20160316 DEBUG Live:
2016-04-28 23:38:30,433 ASG-nginx-images-prod-20160316 DEBUG Emergency:
2016-04-28 23:38:30,433 ASG-nginx-images-prod-20160316 DEBUG LB Unhealthy:
2016-04-28 23:38:30,433 ASG-nginx-images-prod-20160316 DEBUG Checking if needs to launch emergency instances
2016-04-28 23:38:30,433 ASG-nginx-images-prod-20160316 DEBUG Checking if there's any emergency instance to replace
2016-04-28 23:38:30,433 ASG-nginx-images-prod-20160316 DEBUG Checking if it needs to buy spot instances
2016-04-28 23:38:30,488 boto ERROR 400 Bad Request
2016-04-28 23:38:30,488 boto ERROR <?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidGroup.NotFound</Code><Message>The security group 'sg-270fa542' does not exist</Message></Error></Errors><RequestID>718987da-03bd-4753-9ed7-d3dc8072667b</RequestID></Response>
2016-04-28 23:38:30,488 main ERROR EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidGroup.NotFound</Code><Message>The security group 'sg-270fa542' does not exist</Message></Error></Errors><RequestID>718987da-03bd-4753-9ed7-d3dc8072667b</RequestID></Response>
Traceback (most recent call last):
  File "tp.py", line 425, in run
    self.save_money()
  File "tp.py", line 452, in save_money
    self.bid()
  File "tp.py", line 210, in bid
    monitoring_enabled = True)
  File "/usr/lib/python2.7/dist-packages/boto/ec2/connection.py", line 1638, in request_spot_instances
    verb='POST')
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 1186, in get_list
    raise self.ResponseError(response.status, response.reason, body)

Here's my tp.conf file:

{
    "max_price": {
        "t2.micro": "0.200"
    },
    "spot_type": "t2.micro",
    "emergency_type": "t2.micro",
    "max_candidates": 7,
    "instance_name": "thumbor-teste",
    "region": "sa-east-1",
    "placement":  "sa-east-1a",
    "instance_profile_name": "role_baseline",
    "health_check_path": "ping",
    "lower_cpu": 5,
    "upper_cpu": 19,
    "lower_threshold": 1,
    "upper_threshold": 1,
    "tags": {"VREnv":"QA"},
    "user_data_file": null
}

Is there anything wrong in my configuration?
Thanks!