Failed deployments due to throttling about beanstalk-deploy

[dblarons]

Two instances. We use immutable environment updates, which slows our deployment down a bit compared to rolling deploys.

As far as I can tell, it's not necessarily just the frequency (5s) that's the issue, but that the frequency is too much over the ~15-20 minutes it takes for EB to finish our immutable deployment. I'm thinking there's a # of updates per X minutes cap, not just a per second cap.

In any event, we were previously using the command line eb deploy option for our deployment, and that worked fine, so maybe you can just use the same polling period they did?

@dblarons Just published v6 https://github.com/einaregilsson/beanstalk-deploy/releases/tag/v6 which polls less for updates. Try it out and let me know whether it works.

Ok, have published v7 which logs the number of API calls when getting the throttling error. If anyone who gets the throttling error uses it and gets the error, please post back here how many api calls were made. Right now I'm thinking maybe if you have a lot of instances it's calls times instances, so that's why it happens, but it's just a theory for now.

@kceb That's actually a great idea, and what I probably should have been doing from the start. I've uploaded a special branch to try it out, can you try using

uses: einaregilsson/beanstalk-deploy@throttletest

in your deployment script and see whether that works for you? It will allow up to 5 consecutive requests to fail with a throttle error before failing the deployment, and will log a warning if it gets a throttle error. If it does work, please let me see the logs to get an idea of how many times the requests were throttled.

Published v8 which has the throttletest branch merged in. Don't log it as a warning anymore when getting a throttled requests, just log it normally. Closing this issue now, will re-open if there are more problems.

Ok, haven't seen that one. We basically start the deployment, and then poll every 5 seconds to get the status, so the task doesn't finish until the deployment is finished. That doesn't seem like enough requests to get you throttled. You're not using the same IAM key for other stuff at the same time, or doing multiple deploys or anything like that?

[dhimmel]

Has anybody else come across this issue or have any suggestions on how to rectify it?

I also had this once. I gave it time to resolve and haven't seen it again. I didn't think that I should have been hitting the rate limit, but also I'm not exactly sure what's rate limited.

According to this, although it's for EC2: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html#throttling-how

you should have 100 Describe* requests per second allowed, so we shouldn't be anywhere near any limits. No idea what's happening here :/

[owenandrews]

Thanks @einaregilsson @dhimmel! Haven't seen the issue again for a few days now so hopefully it stays away. We're using a dedicated IAM key for just this deployment workflow so it shouldn't be that. Perhaps it's something to do with the way AWS measure request rates? Anyway I'll close this issue for now and let you know if it comes up again.

[dblarons]

I'm seeing this issue every time I deploy. I have a dedicated IAM key that I just created for my workflow.

What does your environment look like? Many instances?

Ok, I'll look into this next week. Either making the polling interval configurable, or by using some kind of exponential backup, where for long deploys it starts polling with longer intervals the longer the deploy is.

[dblarons]

Sounds great — thank you!

[dblarons]

@einaregilsson I tried the new version three times. Once a few hours after you released it (QA), and twice today (Prod, then QA). It timed out the first time, but worked the latter two times. It's possible that there was an issue with Github Actions picking up the new version (as unlikely as that seems).

I think all is good, but I'll keep you posted as I do a few more releases. Thanks for your help!

[kceb]

Facing the same exact issue as @dblarons

I'm on v6 and I consistently get the ##[error]Deployment failed: Error: Status: 400. Code: Throttling, Message: Rate exceeded error on my immutable deploys. Also set up a specific IAM user just for CI.

Screenshot: https://kceb.d.pr/KXadPD

@dblarons are you still experiencing the issue?

@kceb Weird, it's really not making that many calls! Maybe I should add some logging of how many requests have been sent during the deployment.

If you know CloudTrail there's a chance you could see something there: https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-api-throttling-errors/

[dblarons]

@kceb — Yes, I was still having the same issue. I ended up setting up the EB command line to work in my Github Action instead. Code below in case it saves anyone else some time.

    - name: Clone Elastic Beanstalk CLI
      id: clone-awsebcli
      run: git clone https://github.com/aws/aws-elastic-beanstalk-cli-setup.git
            
    - name: Install Elastic Beanstalk CLI
      id: install-awsebcli
      run: ./aws-elastic-beanstalk-cli-setup/scripts/bundled_installer

    - name: Beanstalk Deploy Beta
      id: test-beanstalk-deploy
      run: /home/runner/.ebcli-virtual-env/executables/eb deploy $ENVIRONMENT_NAME --version $VERSION_LABEL --timeout 15

The downside to this method, and why I prefer @einaregilsson's solution, is that it doesn't provide you with any logs until the deploy is finished.

@einaregilsson — I would love to use your project once the throttling is worked out. I'm sorry I can't help out more, I just needed to rush to get this done so I can get back to some other code :) I did spend a while looking through your code and noticed, if I understand it correctly, that you sometimes fire off two API requests at the same time. That was the only thing that stuck out to me that could potentially cause a problem. Other than that your v6 code made sense.

@dblarons I'm thinking about adding a verbose option to the script that logs the number of requests and more extra stuff, to try to understand why this happens. Maybe I need to try to make an environment with immutable updates, see if I can reproduce this. Like any error it's hard to fix when I can't reproduce it :)

Will post back here if I do find something out.

[kceb]

Got the following:
https://kceb.d.pr/pQaPDT

However, some of my merges deploy to multiple environments in parallel. My guess is that is what is hitting the limit.

Is it possible to ignore the 400 limit as an error?

you should have 100 Describe* requests per second allowed, so we shouldn't be anywhere near any limits. No idea what's happening here :/

I don't think I'm anywhere near this limit

[kceb]

During a deploy I have which deploys to two environments in parallel I noticed the warning appear once. It was an immutable deploy so it took around 1 hour total to deploy both in parallel.

##[warning]Request to DescribeEnvironments was throttled, that's 1 throttle errors in a row...

It didn't consider the build failed, so that's good. I'll probably continue using the throttletest branch for now.

I will continue testing and perhaps test on a deploy that deploys to 5 environments