Feature request: Support Capacity Providers about amazon-ecs-plugin

[sithmein]

I believe a minimal first step would be to make the the launchType optional (in com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask). According to the latest AWS docs if you omit the launch type and the capacity provider when submitting a task the default capacity provider is chosen. So you can configure everything in ECS and simply "disable" the launch type in Jenkins. This should be a trivial change. If this is of interest I can even work on a PR for this.

[owengo]

if some want to test/need to launching tasks via the default capacity provider I did a quick hack at: https://github.com/owengo/amazon-ecs-plugin
in "launchType" if you choose "Default Capacity Provider" instead of "EC2" or "Fargate" the "runTask" call will not pass a launchType and so the default capacityProvider for your cluster will be used.
"works for me".

[sithmein]

I did some tests over the weekend but the current API for capacity provider sucks. You can only create capacity providers but neither modify nor delete them. Makes working with them quite hard.
Anyway, once you set the launchType to null in a run task request and have a capacity provider strategy configured it works as expected. I will work in a PR with tests in the next few days. Not sure if I get it done before Christmas, though.

[MikeKroell]

Bump. This code works and if it could be merged into the main, I would avoid having to compiled and run a forked version.

[carlosrodf]

@FnTm I can confirm its working, i had been using a manual build created from the PR that @chathsuom submited for a while in a production setup 👍
I already switched my setup to the newly made release and it is also working fine

[FnTm]

@sithmein that would definitely be a welcome first step. It would be nice to have a more comprehensive support at some point, but I'll definitely take this over nothing 🙂

[webratz]

feel free to submit a PR for that
indeed the capacity providers are interesting to scale your Jenkins clusters in an easier way

[onedumslack]

if some want to test/need to launching tasks via the default capacity provider I did a quick hack at: https://github.com/owengo/amazon-ecs-plugin
in "launchType" if you choose "Default Capacity Provider" instead of "EC2" or "Fargate" the "runTask" call will not pass a launchType and so the default capacityProvider for your cluster will be used.
"works for me".

thank you, I will test this out as I've been manually sending tasks to test scaling.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[owengo]

if some want to test/need to launching tasks via the default capacity provider I did a quick hack at: https://github.com/owengo/amazon-ecs-plugin
in "launchType" if you choose "Default Capacity Provider" instead of "EC2" or "Fargate" the "runTask" call will not pass a launchType and so the default capacityProvider for your cluster will be used.
"works for me".

FYI, we regularly encounter a problem when launching jenkins via ecs-plugin using the "default capacity provider". As explained here: https://forums.aws.amazon.com/thread.jspa?messageID=939879#939879 ,
the containerOverrides passed to the runTask api call ( the "command" is overriden so that the agent knows its name, its secret and where the master is, ) is mysterious lost when the cluster has no provisionned instance. So what happens is that the container is launched but the node can't connect to the master. And the master/plugin does not detect the issue so we have to wait the timeout before a new agent is launched. Usually the second time it works because a provisionned instance is available in the cluster.

[chathsuom]

if some want to test/need to launching tasks via the default capacity provider I did a quick hack at: https://github.com/owengo/amazon-ecs-plugin
in "launchType" if you choose "Default Capacity Provider" instead of "EC2" or "Fargate" the "runTask" call will not pass a launchType and so the default capacityProvider for your cluster will be used.
"works for me".

Can this be incorporated to the main repo? it works.

[juhovan]

I've been using @owengo 's fix for a while and it works perfectly, please include it in the main repo :)

[bryanburke]

I can confirm that lack of capacity provider support also breaks cluster auto scaling. I provide results of my testing with the latest official plugin release on #185.

[chathsuom]

Did some work to support FARGATE_SPOT also #211

[MikeKroell]

Bump on this. This fails to work with the new ECS Capacity Provider as well.

[owengo]

For info:
aws/containers-roadmap#749

Aws just fixed the bug "[ECS] [Bug] containerOverrides not working with capacity provider"

This greatly helps Jenkins with ECS @ Capacity Providers

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[prasanthpola25]

Hi has the update done to main repo for choosing launch type as default capacity provider or leave it? If so which version of ecs plugin has these changes? Appreciate your help!

[FnTm]

The functionality seems to have been introduced by @chathsuom in #211 and released a couple of days ago in 1.39 by @carlosrodf (thanks to both of you!)
Could someone using the plugin with Capacity providers verify if it's working correctly for them? If yes, then I'd like to close this Issue, and any additional enhancements could be discussed in a separate issu.