Possible issue in launching large number of instances and then destroying the cluster about flintrock

[nchammas]

Are you able to reproduce this issue consistently, or is it intermittent?

Is there something special about requesting 25 instances, or do you also see the issue with smaller clusters?

[mkhan037]

I tried to reproduce this issue by using the same command 4 more times. However, this did not happen again. I did encounter issue #340 one of the time.

As for the number 25, there is nothing special other than I was trying to reproduce issue #340 with more than 20 nodes, as it was mentioned there that launching more than 20 nodes can trigger that issue. I did not encounter any issues with small clusters.

[nchammas]

Strange. If this happens to you again and you get some more detail on what the cause is, please post it here.

From your original report, I can see that the initial problem is this:

botocore.exceptions.ClientError: An error occurred (DependencyViolation) when calling the 
DeleteSecurityGroup operation: resource sg-0abb9850b14714211 has a dependent object

I'm guessing that happened only once because the cause is related to AWS resource consistency. Maybe Flintrock deleted something that sg-0abb9850b14714211 depends on, but there was a brief moment when that change didn't propagate sufficiently throughout AWS. So when Flintrock went to delete sg-0abb9850b14714211 itself, we got this dependency violation.

That's just a guess. But if that was the issue, then I suppose a general solution would be to adjust some boto settings to retry operations more times before failing (though I don't think those settings apply to DependencyViolation errors), or perhaps to add waiters on specific operations that are known to take time to propagate out, like deleting a security group dependency (e.g. another security group, or a security group rule).

[mkhan037]

Strange. If this happens to you again and you get some more detail on what the cause is, please post it here.
If I reencounter this issue, I will surely let you know. Thanks again for this awesome tool.

[nchammas]

Thanks for the report!