turnerlabs / fargate Goto Github PK

We have added secretes to an existing task definition and once deploying with with the cli, it appears to blow them away, breaking the service requiring them.

Is this in the road map at all?

In an effort to support Fargate tasks with multiple containers, there are a number of use cases that we'll need to address. I will try to lay out a proposal in this issue. Feedback welcome.

1. add/update/delete containers in a task definition

First some quick background. Our philosophy has been to use different tools to separate infrastructure and application concerns. We tend to use Terraform (or Cloud Formation) for provisioning the initial infrastructure and this tool for managing ongoing application concerns on top, like docker images, environment variables, and secrets. When it comes to creating an app, we've relied on fargate-create and terraform templates to initialize the ECS task definition with a placeholder container. This tool then takes over and uses docker and docker-compose as simpler abstractions on top of task definitions by revising them over time with app concerns (images, envvars, secrets) and deploying them to ECS services in a single command. Using docker abstractions also creates more of a seamless transition between running an app locally then deploying the same configuration to the cloud.

When it comes to configuring multiple containers in a task definition, unfortunately, docker abstractions aren't really sufficient. Task definitions provide a superset of what's in docker/docker-compose with constructs like container dependency graphs with conditions and EFS configuration, for example. This means that I think we should allow the user to edit a task definition directly (which is where our abstraction will leak). Rather than requiring a user to shift over to a completely different tool, we could introduce a command that would allow them to export their service's current task definition to a json file which can then be edited and directly re-deployed.

fargate service describe -o json > task-definition.json

fargate service deploy -f task-definition.json

This could come in handy for a number of scenarios by giving the user full control in times when they need to change something that isn't available in the docker abstractions. For example, as previously mentioned, they could leverage container dependencies to control exactly how their containers relate to and depend on each other for startup and shutdown. Again, this construct is not available in docker and is best expressed in task definition json.

2. deploy multiple app containers (i.e.; CI/CD)

Once all the containers have been added/updated/deleted in a task definition and deployed to a service (fargate service deploy -f task-definition.json), the service deploy command should then be able to deploy the app level concerns (images, envvars, and secrets) to one or many containers in a service. For deploying changes to a single container, we could add a --container flag which specifies the name of the container in the task definition to deploy to.

fargate service deploy --image <image> --container app

For deploying to multiple containers in a single command, we could deploy a docker-compose.yml file where the service names would map to container names in the task definition. For backwards compatibility, if there's only 1 container in the compose file, it could deploy to the first container in the task definition without requiring the name mapping.

fargate service deploy -f docker-compose.yml

3. set/unset envvars for multiple containers

As far as setting and unsetting environment variables, we could add a --container flag similar to service deploy --container.

fargate service env set -e KEY=VALUE --container app

If the flag is not specified, the target container is the first one in the list (as it is today, for backwards compatibility).

4. show info for multiple containers

The service info command is already sort of a task-level view of a service. We could add a new service containers command to show information about multiple containers. Something like:

fargate service containers

CONTAINER:   app
IMAGE:       1234567890.dkr.ecr.us-east-1.amazonaws.com/app:1.0
ENVIRONMENT VARIABLES:
  HEALTHCHECK=/health
  PORT=8080
  LOG_LEVEL=debug
  SECRET=/var/secret/my-secr

CONTAINER:   secrets-sidecar
IMAGE:       quay.io/turner/secretsmanager-sidecar:0.1.0
ENVIRONMENT VARIABLES:
  SECRET_ID=my-secret
  SECRET_FILE=/var/secret/my-secret

5. show logs for multiple containers

If all containers are setup to log to the same log group, we can use the existing logs command and grep for the specific container we're interested in.

fargate service logs | grep "ecs/app"

If we wanted to make this easier, I suppose we could add a --container flag that could do the filtering.

fargate service logs --container app

How does this sound? Thoughts?

This is a proposal to enhance support for scheduled tasks by adding the ability to update cloudwatch event rules, targeting new task definition revisions.

fargate events target --rule <rule> --revision <revision>

or --rule can be omitted if the rule name is added to fargate.yml...

cluster: my-app-dev
service: my-app-dev
task: my-app-dev
rule: my-app-dev

fargate events target --revision <revision>

So a CI/CD system could build new apps (#20) (docker images and environment variables)...

REVISION=$(fargate task register -i <image> -e FOO=bar)

...then "deploy" them (cause the next event rule invocation to run the new version)

fargate events target --revision ${REVISION}

When you do a fargate service help it says --nocolor Disable color output, but the actual flag is --no-color.

I was trying to use fargate service deploy with both the image and revision flags and it kept just erroring out to the usage message. Nowhere did it say that I can't use those two options together. Since the command was buried in a CodeBuild yaml it took me a while to figure out why it was failing.

Now that tags are supported, we should propagate existing tags when copying task definitions.

It looks like service env unset doesn't support secrets and removing them from a compose yml then service deploy -f doesn't remote them from the task definition.

The goal with this would be to support the ability to register new task definitions with "application" type concerns like images and environment variables, in much the same way that fargate service deploy iterates on existing task definitions (initially created by some other tool, like terraform). It would be nice if it supported passing the data via CLI args or a docker-compose.yml file (same as fargate service deploy). So for example:

fargate task register -f docker-compose.yml

OR

fargate task register -i <image> -e <name>=<value>

Using this approach, CI/CD pipelines could push out new scheduled tasks with something like:

# register task definition with new docker config
REVISION=$(fargate task register -i <image> -e FOO=bar)

# deploy new build
aws lambda update-function-configuration \
  --function-name my-app-task-dev \
  --environment "Variables={REVISION=$REVISION}"

Once AWS supports fargate as a native cloudwatch event rule target, then obviously the use of lambda in the example above would no longer be needed.

It would be nice to have another global config for the task name so that it doesn't have to be specified at the command line. Something like this in fargate.yml:

cluster: my-app-dev
service: my-app-dev
task: my-app-task-dev

from awslabs#40

It would be cool to add support for podman-compose! Podman is far more secure than docker and is a drop-in replacement as well which is an added bonus

The documentation is a bit old and could be updated to reflect the newer go changes such as using modules rather than dep.

Would be nice to support a config file, envars, and cli args for specifying the cluster and service.

I would love to help with this, but I don't know much about Golang. I tried and I have reached a limit of what I know and I can do. So I am opening this issue so someone with better understanding of Golang can fix it. Seems to be a fairly easy fix I think but requires understanding about some Golang concepts that I am not familiar with.

Description
We are using dockerhub instead of ECR for ease of use (we have same service un multiple AWS regions, and ECR is per-region only). AWS Fargate supports private registries as reported here

For this, we have to specify a repositoryCredentials in the containerDefinition (doc).

Problem
The problem is that when deploying the new image, the new TaskDefinition release does not include the repositoryCredentials that was set in the previous release.

Command
ncd service deploy --service $servicename --cluster $clustername -i $dockerhub_image --region $regions -v

Result
Service gets deployed, but the created task definition doesn't contain repositoryCredentials.

Observations
Using the -v option, I can see that the AWS API response for the previous task definition that is obtained from here includes the proper repositoryCredentials but when I print to console the dtd variable using console.Info("--- Resp: %#v", dtd), repositoryCredentials is not there.

Expected Result
The expected result should be that the repositoryCredentials parameter should be carried forward to the new registered TaskDefinition

I have multiple AWS profiles saved in my ~/.aws/credentials file, but the fargate command does not allow me to specify a --profile parameter like the awscli commands allow me to do.

If I set the region via:

aws configure set region "us-west-2"

Does fargate cli pull the region from aws configure?

As a user of the fargate command I would like to be able to deploy my different environments using an inheritance/override model that is allowed with the docker-compose command.

eg: fargate service deploy --service foo --cluster bar -f docker-compose.yml -f docker-compose.dev.yml

#57

Perhaps this is down to me not understanding the tool correctly but once I switched to the CLI's fargate deploy from the ordinary terraform deployment, I noticed that old task definition versions stay around.

In practice this is not a huge problem apart from one issue: when you stop a container (for example to force a reload due to a secret having changed) then an old task definition is started.

Is this behaving as expected?

Is there a CLI command to remove or deactivate the old task definitions?

When running the following command with docker-compose version 1.27.2, build 18f557f9 installed, the following error occurs.

fargate service deploy -f docker-compose.yml

[!] Please indicate which docker container you'd like to deploy using the label "aws.ecs.fargate.deploy: 1"

Not seeing values like stopTimeout make it to the new task definition revision when running fargate service deploy.

I just tried to deploy a service to AWS Fargate in the eu-central-1. It doesn't work because of this limitation:
https://github.com/turnerlabs/fargate/blob/master/cmd/root.go#L47

I can't see why this should not work in other zones.

Currently, the fargate service info command displays some recent, basic ECS events (that come from the describe-services api). These events are limited to showing tasks starting, stopping, and registering/deregistering from load balancers, and don't give you as much detail as the console does, particulary when things go south. A typical deployment, for example:

Events
[2018-11-27 19:15:36 +0000 UTC] (service foo) has reached a steady state.
[2018-11-27 19:15:16 +0000 UTC] (service foo) has stopped 2 running tasks: (task a2dee190-4da5-4144-9c99-5f6ad1187b47) (task 1c11a717-a7d2-439c-88ca-b7195541e3ec).
[2018-11-27 19:14:56 +0000 UTC] (service foo) has begun draining connections on 2 tasks.
[2018-11-27 19:14:56 +0000 UTC] (service foo) deregistered 2 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/foo/882e246684f065c1)
[2018-11-27 19:14:16 +0000 UTC] (service foo) registered 2 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/foo/882e246684f065c1)
[2018-11-27 19:13:27 +0000 UTC] (service foo) has started 2 tasks: (task 585443aa-6654-4a18-b7a8-c463de1beb2e) (task 4c6997da-0db0-4380-bc47-90e0d8654444).
[2018-11-27 17:27:41 +0000 UTC] (service foo) has reached a steady state.

I'm proposing that we add a fargate ecslogs command which can parse the full ECS event stream from CloudWatch logs (assuming it's been configured via a cw event rule/lambda which will soon be supported in the terraform-ecs-fargate template). This would expose the full, rich event log which includes more information to help troubleshoot and understand what ECS is doing. A typical deployment might look something like this:

$ fargate ecslogs

TIME                  REVISION  IMAGE PULL  IP             DESIRED STATE   CURRENT STATE
2018-11-27T11:00:41Z  2                                    RUNNING         PROVISIONING
2018-11-27T11:00:41Z  2                     10.65.234.123  RUNNING         PENDING
2018-11-27T11:00:41Z  2         30 seconds  10.65.234.123  RUNNING         RUNNING

Another example, trying to deploy an image tag that hasn't been pushed to a registry:

$ fargate ecslogs

TIME            REVISION   IMAGE PULL  IP         DESIRED STATE   CURRENT STATE                   
5 minutes ago   3                                 RUNNING         PROVISIONING                             
5 minutes ago   3                                 RUNNING         PENDING                         
5 minutes ago   3                                 STOPPED         DEPROVISIONING                                                      
1 minute ago    3                                 STOPPED         STOPPED (CannotPullContainerError: API error (404): manifest for 123456789.dkr.ecr.us-east-1.amazonaws.com/foo:0.1.0 not found)  
1 minute ago    3                                 RUNNING         PROVISIONING
1 minute ago    3                                 RUNNING         PENDING
1 minute ago    3                                 STOPPED         DEPROVISIONING
1 minute ago    3                                 STOPPED         STOPPED (CannotPullContainerError: API error (404): manifest for 123456789.dkr.ecr.us-east-1.amazonaws.com/foo:0.1.0 not found)

Auto-scaling actions would also show up here.

eu-west-3 is a valid region but is not supported by fargate cli.

fargate task describe --region eu-west-3

[!] Invalid region: eu-west-3 [valid regions: us-east-1, us-east-2, us-west-1, us-west-2, ca-central-1, eu-west-1, eu-west-2, eu-central-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, ap-northheast-2, ap-south-1]

Occasionally when running fargate service ps the result is incorrectly No Tasks Found. This may also explain why the task list is missing from fargate service info occasionally.

My hunch is that the aws sdk call is responding with an error, but it isn't being handled properly in the code.

It would be cool to add the ability to deploy a particular task definition revision to a service. This would provide true 12-factor rollbacks where a previous docker image and environment variable configuration could be re-deployed exactly as it was.

Particularly now that we have:

REVISION=$(fargate task register -i <image> -e <KEY=VAL>)

example:

fargate service deploy --revision 42

When using the fargate service deploy command, it attempts to copy the previous task definition but does not maintain the original runtimePlatform configuration. This means it becomes null and Fargate will default the task to run as LINUX+amd64, which is not always the case.

Per the instructions:

$ fargate service env unset -h
Unset environment variables

Unsets the environment variable specified via the --key flag. Specify --key with
a key name multiple times to unset multiple variables.

Usage:
  fargate service env unset --key <key-name> [--key <key-name>] ... [flags]

Flags:
  -h, --help              help for unset
  -k, --key stringSlice   Environment variable keys to unset [e.g. KEY, NGINX_PORT]

I should be able to run this:

$ fargate service env unset -k TEST_SET -k TEST_SET4
[i] Unset cartoon-games-director-dev environment variables:
[i] - TEST_SET
[i] - TEST_SET4

It appears to work, but the variables are not removed from the newly created revision. Using "--key" doesn't make a difference.

$ fargate service env unset --key TEST_SET3 --key TEST_SET4 --key TEST_SET
[i] Unset cartoon-games-director-dev environment variables:
[i] - TEST_SET3
[i] - TEST_SET4
[i] - TEST_SET

The only way to remove multiple keys is one by one:

$ fargate service env unset -k TEST_SET
[i] Unset cartoon-games-director-dev environment variables:
[i] - TEST_SET
$ fargate service env unset -k TEST_SET4
[i] Unset cartoon-games-director-dev environment variables:
[i] - TEST_SET4

Feature Request: Fargate cli only allows updates to the main (first) container. There is no way to specify updates to sidecar containers

Hi,

We have come across a weird issue with fargate cli. We are working to store all of our environment variables using context service. To do this we need to add a task definition secret. This command will add said secret:

fargate service env set -c wme-gep-drupal -s wme-gep-drupal-dev --secret CONTEXT_KEYS=arn:aws:ssm:us-east-1:172852125925:parameter/wme-gep-context-service/dev

We however keep seeing this secret disappear. After many days of diagnosing we came across the issue. When someone runs a fargate service update command it will remove that task definition.

Here is command that was used:

fargate service update --service wme-gep-drupal-dev --cluster wme-gep-drupal --cpu 2048 --memory 4096

Please let us know if we can somehow get this patched as this could bring down our system if the variables are no longer present.

Best Regards,
Mike

to reproduce create a task with 2 container definitions. Add a firelensConfiguation to one. Try to update the primary app. result:

ClientException: When awsfirelens log driver is specified in log configuration, a firelens configuration object must be configured in one of the containers.
	status code: 400, request id: eb51c4dd-5eda-4005-8fcf-3b612a4864cf

something to do with how the TaskDefinitionOutput struct is being passed around. The base task is defined below:

  "containerDefinitions": [{
    "name": "app",
    "image": "whatever/image",
    "cpu": 0,
    "portMappings": [{
      "containerPort": 80,
      "hostPort": 80,
      "protocol": "tcp"
    }],
    "essential": true,
    "environment": [ {
      "name": "HEALTHCHECK",
      "value": "/_healthcheck"
    }, {
      "name": "LOG_LEVEL",
      "value": "info"
    }],
    "mountPoints": [],
    "volumesFrom": [],
    "secrets": [{
      "name": "MONGO_PASSWORD",
      "valueFrom": "arn:aws:ssm:us-east-1:287171731453:parameter/varys/dev/MONGO_PASSWORD"
    }, {
      "name": "MONGO_USER",
      "valueFrom": "arn:aws:ssm:us-east-1:287171731453:parameter/varys/dev/MONGO_USER"
    }],
    "logConfiguration": {
      "logDriver": "awsfirelens",
      "options": {
        "Name": "Http",
        "Host": "listener.logz.io",
        "URI": "/?token=secretSTring&type=fargate",
        "Port": "8071",
        "tls": "on",
        "tls.verify": "off",
        "Format": "json_lines"
      }
    }
  }, {
    "name": "logzio-log-router",
    "image": "amazon/aws-for-fluent-bit:latest",
    "essential": true,
    "firelensConfiguration": {
      "type": "fluentbit",
      "options": {
        "enable-ecs-log-metadata": "true"
      }
    },
     "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/fargate/service/varys-dev",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
    }
  }],
  "family": "varys-dev",
  "taskRoleArn": "arn:aws:iam::287171731453:role/varys-dev",
  "executionRoleArn": "arn:aws:iam::287171731453:role/varys-dev-ecs",
  "networkMode": "awsvpc",
  "volumes": [],
  "placementConstraints": [],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "256",
  "memory": "512"
}

awslabs#10