josmo / drone-ecs

Drone plugin for triggering Amazon EC2 Container Service (ECS) deployments

Hi there 👋

• 🔭 I’m currently working on mostly HA services (usually with Spring Boot-Java/Kotlin, GoLang, NodeJS, and more) in various cloud providers with K8s, Terraform, helm, Flux, Flagger, etc.
• 🌱 I’m currently learning Spanish as well as focusing on improving my surfing
• 👯 I’m looking to collaborate on anything that helps people
• 🤔 I’m looking for help with recruiting amazing developers
• 📫 How to reach me: josmo on Keybase, lindyblues on twitter

drone-ecs's Issues

Deploying to FARGATE

Having troubles deploying a task on a Fargate Cluster, it complains Task definition does not support launch_type FARGATE

Also when I specify network_mode to awsvpc it complains Error response from daemon: network awsvpc not found

Merge existing task definitions

More of a feature request, apologies if not the right place to post.

We're migrating from Jenkins to drone and it's great, but in regards to publishing to ECS it would be good if could be a bit more dynamic. We use cloud formation to bring up all our services so the ECS service, definition and task names are all dynamic along with the environment variables we inject for things like database IP addresses etc. Currently we have a script in jenkins that pulls down the current service definition as a template and replaces the values for the image to be used. This way we don't need to hard code any of the environment variables into jenkins that could change through cloud formation.

Also it takes the environment (eg production) and finds the appropriate cluster / service first rather than needing to hard code those names too, which may change. eg an api cluster might be called production-api-JH67JGSJK, with a random value appended to the base name production-api through the cloud formation process and a db cluster might be production-db-HGDTY13TH.

It's just a simple bash script using the aws cli, first we pull down all the services and grep on the environment, then on the service ie db or api. This way we can then drill in to the service and find the latest task def for say production-api for a cloud formation built service

So I was wondering if this sort of functionality could be considered to be added. Rather than specifying a fixed service name or family, could it be derived from a substring. Then pull the existing definition and merge any new settings like cpu, mem, env vars, image name etc if they are specified in the yml otherwise use the existing tasks settings. So you only need to specify in the drone.yml what has changed.

I've attached an example bash script, this script is for a cluster with a base name of ServicesCluster which contains a service with a base name that contains the word 'Dashboard'. The service only has one task so it derives the current task from that, but it could easily handle finding a specific task in the service as well if you passed in a base name for the task.

Thanks in advance for any consideration and review

Cheers
Paul

jenkins_build.txt

firelensConfiguration support?

Is firelensConfiguration supported?

deploy fail with image peloton/drone-ecs

I use image: peloton/drone-ecs to deploy.
But recently I met some problem when deploying.

This is fail info below:

1 | Drone AWS ECS Plugin built
2 | 2019/07/04 08:26:29 InvalidParameter: 1 validation error(s) found.
3 | - missing required field, UpdateServiceInput.NetworkConfiguration.AwsvpcConfiguration.Subnets.

and this is deploy part of my .drone.yml:

  deployDevProxy:
    image: peloton/drone-ecs
    region: ap-northeast-1
    service: ****-frontend-dev
    container_name: ****-frontend-proxy
    family: ****-frontend-dev
    cluster: ****-ECS-dev
    task_role_arn: ****-frontend-task-role
    docker_image: ********.dkr.ecr.ap-northeast-1.amazonaws.com/****-frontend
    tag: develop
    environment_variables:
      - NODE_ENV=development
      - API_ENDPOINT=https://****/api
      - BUILD_TAG=${DRONE_BRANCH}-${DRONE_BUILD_NUMBER}
    cpu: 256
    memory: 256
    desired_count: 1
    deployment_configuration:
      - 50 200
    log_driver: awslogs
    log_options:
      - awslogs-group=****-frontend
      - awslogs-region=ap-northeast-1
      - awslogs-stream-prefix=/dev
    port_mappings:
      - 0 8080
    when:
      branch: [develop]

I've tried to use specific version image: peloton/drone-ecs:1.0.6 for building, it's okay in process.
Could I get some suggestions to trace/correct this error?

AWS access_key and secret_key results in InvalidParameterException

Hi, I read the docs and also checkedmain.go and plugin.go.

I've created two organization secrets in drone, aws_access_key_id and aws_secret_access_key, I'm already using them in many pipelines/steps, but it seems like they don't work with this plugin.

My .drone.yml snippet

  - name: deploy_to_ecs
    image: pelotech/drone-ecs
    pull: always
    environment:
     AWS_ACCESS_KEY:
       from_secret: aws_access_key_id
     AWS_SECRET_KEY:
       from_secret: aws_secret_access_key
    settings:
      cluster: dev-cluster
      service: dev-service
      deployment_configuration: 100 200
      docker_image: 1234567890.dkr.ecr.eu-west-1.amazonaws.com/some-image
      region: eu-west-1

According to the code, I can see that if I don't pass access_key and secret_key, that's ok, as long as I set AWS_ACCESS_KEY and AWS_SECRET_KEY.

Here's the error I get when using the snippet above

Status: Image is up to date for pelotech/drone-ecs:latest
--
4 | Drone AWS ECS Plugin built
5 | 2020/06/28 14:53:10 InvalidParameterException:

According to the plugin, it should work - main.go - Ref

	app.Flags = []cli.Flag{
		cli.StringFlag{
			Name:   "access-key",
			Usage:  "AWS access key",
			EnvVar: "PLUGIN_ACCESS_KEY,ECS_ACCESS_KEY,AWS_ACCESS_KEY",
		},
		cli.StringFlag{
			Name:   "secret-key",
			Usage:  "AWS secret key",
			EnvVar: "PLUGIN_SECRET_KEY,ECS_SECRET_KEY,AWS_SECRET_KEY",
		},
...
func run(c *cli.Context) error {
	plugin := Plugin{
		Key:                          c.String("access-key"),
		Secret:                       c.String("secret-key"),
...

And a reference of where the creds are fetched - plugin.go - Ref

	if len(p.Key) != 0 && len(p.Secret) != 0 {
		awsConfig.Credentials = credentials.NewStaticCredentials(p.Key, p.Secret, "")
	}
	awsConfig.Region = aws.String(p.Region)


	var svc *ecs.ECS
	sess := session.Must(session.NewSession(&awsConfig))

[feature] Add support for assignPublicIp

We need support to assign public IPs, since ECS cannot start new tasks now:

CannotPullContainerError: Error response from daemon: Get https://xxx.dkr.ecr.eu-central-1.amazonaws.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_AwsVpcConfiguration.html

Bug - Passing more than one command as entrypoint override

Due a pointer wrong reference, when I try to pass more than one command, this happens:

PR> #50

[Feature] Support for privileged mode

We need support for flag privileged, that allows running tasks in privileged mode (applicable only for EC2 launch type)

ValueFrom field in environment variables

It would be great (and fairly straightforward, I'd believe?) if we could create the container definition with environment variables that support the valueFrom field.

Something like:

environment_variables_from:
  - DB_PASSWORD=<ssm_parameter_store_key_arn>

From the AWS console:

I personally use AWS SSM Parameter Store for some settings and would prefer to keep using it rather than relying on Drone's internal secret store for the application secrets.

What do you think?

Support for LogConfiguration

Hi,

Is it possible to add a LogConfiguration support for the update service part?

Labels does not work as documented.

Hello,
Im having trouble getting drone-ecs add the labels, ive tried with different formatting and style of strings, but nothing seem to be add to the output of the task definition i can see i drone log from the build step.

input (copied from the docs)

    labels:
      - traefik.frontend.rule=Host:my.host.gov

plugin output

DockerLabels = {

}

Full build step:

  ecs-deploy-web:
    image: peloton/drone-ecs
    cluster: "clustername"
    service: service-name
    container_name: web
    docker_image: xxx.amazonaws.com/web
    tag: ${DRONE_BRANCH}
    desired_count: 1
    deployment_configuration: 100 200
    memory_reservation: 1024
    memory: 1024
    family: web-${DRONE_BRANCH}
    labels:
      - traefik.frontend.rule=Host:my.host.gov
    log_driver: "awslogs"
    log_options:
      - awslogs-group=loggroup

[feature] add ability to create new service if it doesn't exist

The requirement to have the ECS service running before using the plugin brings a couple of problems, specifically the duplication of configuration across drone and the initial creator application. It would be helpful if the plugin had the ability to create the service if it doesn't exist on the cluster, so the plugin can be used for the full service lifecycle.

Allow setting desired count to 0

The plugin forbids setting the desired count parameter to zero here:

drone-ecs/plugin.go

Line 188 in 5a26ad4

if p.DesiredCount != 0 {

This prevents me from scaling down an existing ECS deployment. Is there a specific reason why this is the case?

Comma in env variables causes panic

Hi,

Whenever I try to use commas in the environmental variables the plugin panics. I've tried wrapping the value with " but it does not help. I've noticed the same issue in two different code projects. If I put the value in a env secret then it works fine.

  - name: deploy-production
    image: josmo/drone-ecs:latest
    ...
    settings:
      environment_variables:
        - DRONE_USER_CREATE=username:joakim,admin:true

Drone AWS ECS Plugin built

5 | panic: runtime error: index out of range [1] with length 1
6 |  
7 | goroutine 1 [running]:
8 | main.(*Plugin).Exec(0xc0000aab00, 0xc0001b2180, 0xa2208b)
9 | /go/src/github.com/tallinger/drone-ecs/plugin.go:212 +0x4155
10 | main.run(0xc0000a8420, 0x0, 0xc0000d7190)
11 | /go/src/github.com/tallinger/drone-ecs/main.go:278 +0x1347
12 | github.com/urfave/cli.HandleAction(0x962780, 0xa44bb0, 0xc0000a8420, 0xc0000a8420, 0x0)
13 | /go/pkg/mod/github.com/urfave/[email protected]/app.go:514 +0xbe
14 | github.com/urfave/cli.(*App).Run(0xc0001c2000, 0xc0000761b0, 0x1, 0x1, 0x0, 0x0)
15 | /go/pkg/mod/github.com/urfave/[email protected]/app.go:274 +0x5f6
16 | main.main()
17 | /go/src/github.com/tallinger/drone-ecs/main.go:230 +0x1ef9

Any ideas what is wrong? I had a look at row 212 in plugin.go but I do not see anything obvious wrong?

Unable to use/inject secrets as environment variables

The main reason I opted for this plugin vs the other ecs plugin was that it appeared more complete, and that it clearly indicates the ability to inject secrets as environment variables.

I am, however, completely unable to do so. Ideally I want to pull secrets from aws secrets manager, which I have everything necessary configured on the drone server, and can pull secrets via the cli tool. However, within config for this plugin, whether I try to use the secret_environment_variables or just environment_variables and whether the secret is coming from the amazon secrets external secrets source, or inbuilt drone repo secrets config, no matter what I try the value is always an empty string.

---
kind: secret
name: STG_NODE_ENV
get:
  path: xxxx/staging
  name: NODE_ENV

That ought to be available as STG_NODE_ENV so I would think I should be able to do either of these:

environment_variables:
  - NODE_ENV=${STG_NODE_ENV}

or:

secret_environment_variables:
  - NODE_ENV=STG_NODE_ENV

Neither work, always results the same getting defined in the ecs task definition:

Environment: [{
Name: "NODE_ENV",
Value: ""
}],

I'm dead in the water if I can't populate environment variables!

Container name is not configurable but defined by convention

If we check line https://github.com/drone-plugins/drone-ecs/blob/master/main.go#L115 the container name in the Task Definition is not configurable but conventionally set to {family-name}+'container'.
This is not documented and we should make it configurable as it might conflict with initial configurations especially when the service is behind an Application Load Balancer.

Ability to add placement constraints to ecs task definitions

Would be great if we had the ability to specify placement constraints during task definition registration. (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-constraints.html)
I've added the functionality on our fork of the repo. Would be happy to raise a PR if this is a feature that would make sense.

InvalidParameterException when using plugin with fargate

We have an issue when running the plugin. We always get the exception "InvalidParameterException" although all parameters have been specified. The Task definition will be created by a cloud formation script and therefore must not be created by the plugin. However we still get the InvalidParameterException and we can not figure out how to fix it.
This is our config:

- name: dev:fargate:deploy
   image: pelotech/drone-ecs
   settings:
    region: us-east-1
    family: IndiAuthTaskDefinition
    docker_image: xxxx.dkr.ecr.us-east-1.amazonaws.com/xxxxx-backend
    tag: latest
    service: IndiauthService
    cluster: IndiauthCluster
    container_name: IndiauthContainerDefinition
    deployment_configuration: 50 200
    access_key:
        from_secret: DEV_AWS_ACCESS_KEY_ID
    secret_key:
      from_secret: DEV_AWS_SECRET_ACCESS_KEY
    

Feature: service_network_security_groups from secrets

Hi Team

Would be nice to have a flag that would allow loading service_network_security_groups from secrets

This could be done by a new flag eg: **secret_**service_network_security_groups or similar

Our case:
we have a thousand lines of configuration for each QA environment, the only one difference between each QA are service_network_security_groups, would be nice to deal with duplication, then we can use a trick to promote

secret_service_network_security_groups:

• sg_common_name
• sg_${DRONE_DEPLOY_TO}_name

Support for multi-container tasks

Hello, currently only one container is supported by the Plugin, as hardcoded in https://github.com/josmo/drone-ecs/blob/master/plugin.go#L252

Any chance we're going to implement multi-container definitions soon?

Thanks

deploy fail, claiming missing region, it's clearly specified.

While attempting to deploy to ECS, I am unable to get beyond a "MissingRegion" error, yet the region clearly is defined. Have also tried defining it as an environment variable PLUGIN_REGION, as I see that's the other thing it looks for, neither work.. this is a major show stopper.

Have tried latest and 1.0.6 releases of the plugin, both fail the same way.

config:

steps:
  - name: Deploy to ECS
    image: peloton/drone-ecs:1.0.6
    region: us-east-1
    family: ui-backend-dev
    service: ui-backend-dev
    tag: latest
    task_role_arn: arn:aws:iam::xxxxxxxx:role/ecsTaskExecutionRole
    log_driver: awslogs
    log_options:
      - awslogs-group=/ecs/ui-backend-dev
      - awslogs-region=us-east-1
      - awslogs-stream-prefix=ecs
    port_mappings:
      - 3333 3333
    compatibilities:
      - EC2
      - FARGATE
    memory: 2048
    cpu: 1024
    desired_count: 2
    deployment_configuration: 100 200
    docker_image: xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/ui-backend-dev
    secrets:
      - AWS_SECRET_KEY
      - AWS_ACCESS_KEY

Add support for setting Fargate network configuration

When deploying to Fargate you can supply networking configuration to dictate the subnets and security groups used by the tasks. If this is not supplied the defaults are used, which works but is not ideal. It would therefore be advantageous to add support for supplying network configuration when deploying to ECS.

Problem with mount points

Hi I havent found a way to use volume/mount points. Firstly I though that was using the current definition to modify all the parameters specified on the plugin, but it seems to create a new definition from scratch, how can I make this plugin able to add volumes?

Thanks!

NoCredentialProviders: no valid providers in chain. Deprecated.

Any idea why im getting this? Thanks!

latest: Pulling from pelotech/drone-ecs
Digest: sha256:2a3dd10ee0408a06384c79772a9b13e3cff8c92e0ac6346c2ffcfa3a53193d95
Status: Image is up to date for pelotech/drone-ecs:latest
Drone AWS ECS Plugin built
2020/02/27 22:13:05 NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Assume different user role

Hi,

Would it be possible to add support for assuming a different user role?

Key/Secret -> login -> assume role -> create task. Something like this in settings:

user_role_arn: arn:aws:iam::012345678901:role/APP_TEST_QA_SERVICE
task_role_arn: arn:aws:iam::987654321098:role/ecs-executionrole-qa

From command line:
$aws sts assume-role --role-arn "${USER_ROLE_ARN}"

Reference: https://github.com/shipt/drone-ecs-deploy/blob/282d911c1035c88f76247a0ef850faff36fd7c4f/ecs-deploy#L108

status code: 400 returned does not cause build to fail

The following error isn't causeing the build job to fail, it's still completes with a status of successful.

Drone AWS ECS Plugin built at 2016-01-26 08:59:19 UTC
ServiceNotFoundException: Service not found.
status code: 400, request id: 8eacbfc6-cd58-11e5-8bb2-4558b2059ecf

As per bradrydzewski comment in gitter

bradrydzewski 09:01
it looks like this error is being triggered https://github.com/drone-plugins/drone-ecs/blob/master/main.go#L178
which does not result in an exit 1 but probably should

Cheers
Paul

Dockerhub - You have reached your pull tate limit

Hello,

Dockerhub 20.11.2020 has announced anonymous pulling rate limit.
https://www.docker.com/increase-rate-limits

For building our images we are using docker hub images. We would like to have a possibility to add Docker Hub credentials into ECR pipeline step to not see this error anymore.