memosstilvi / emr-cost-calculator Goto Github PK

python2.7 emr_cost_calculator.py cluster --region=us-east-1c --cluster_id=j-WDGRM50XNDZQ --aws_access_key_id= --aws_secret_access_key=

enable: print(sys.executable) on 34,35,36 on emr_cost_calculator.py
execution output:-

/usr/local/opt/python/bin/python2.7
/usr/local/opt/python/bin/python2.7
/usr/local/opt/python/bin/python2.7
[INFO] Retrieving cost in region us-east-1c
Traceback (most recent call last):
File "emr_cost_calculator.py", line 238, in
print calc.get_cluster_cost(args.get('--cluster_id'))
File "/usr/local/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/local/lib/python2.7/site-packages/retrying.py", line 206, in call
return attempt.get(self._wrap_exception)
File "/usr/local/lib/python2.7/site-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/local/lib/python2.7/site-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "emr_cost_calculator.py", line 132, in get_cluster_cost
instance_groups = self._get_instance_groups(cluster_id)
File "emr_cost_calculator.py", line 176, in _get_instance_groups
groups = self.conn.list_instance_groups(cluster_id).instancegroups
AttributeError: 'NoneType' object has no attribute 'list_instance_groups'

module version : 'retrying==1.3.3' 'docopt==0.6.2' 'docutils==0.12' 'pyyaml==3.12'

six version is update on my mac 'six==1.10.0'
boto version is update to 'boto==2.39.0'
pyyaml version is update to 'pyyaml==3.12'

i'm unable to get the issue, can you please review the stack trace. thanks in advance.

ERROR on google: https://julien.danjou.info/blog/2015/python-retrying

./emr_cost_calculator.py cluster --region=eu-west-1 --cluster_id=j-GCYS9FTHGE3I
[INFO] Retrieving cost in region eu-west-1
Traceback (most recent call last):
File "./emr_cost_calculator.py", line 266, in
print calc.get_cluster_cost(args.get('--cluster_id'))
File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 206, in call
return attempt.get(self._wrap_exception)
File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "./emr_cost_calculator.py", line 145, in get_cluster_cost
instance_groups = self._get_instance_groups(cluster_id)
File "./emr_cost_calculator.py", line 203, in _get_instance_groups
inst_group.price = prices[group.instancetype]['ec2'] +
KeyError: u'm4.xlarge'

https://github.com/memosstilvi/emr-cost-calculator/blob/master/emr_cost_calculator.py#L190

Using the BidPrice means the value of the cluster is reported as being the maximum bid price specified by the user times the lifetime in hours of those instances (https://github.com/memosstilvi/emr-cost-calculator/blob/master/emr_cost_calculator.py#L64)

This is incorrect because the actual cost of the instance per hour is determined by the Spot Instance Market, which fluctuates (in this case) between points of the spot instances price floor and the user's max bid price. A four instance EMR cluster that runs for 10 hours at a bid price of $0.37/hour would be returned by this calculator as having cost $14.8 ($0.37 * 4 * 10). However in actuality the cost would have been ($0.20 * 4 * 8) + ($0.37 * 4 * 2) = $9.36 (~36.8% cost difference) because the actual cost per instance was only $0.20/hour for 8 hours and the max bid price of $0.37/hour for 2 hours. Also, EMR nodes have an added fixed EMR node fee per hour (in addition to the EC2 instance cost, whether on demand or spot).

Checked the script and found bid price is used.