mediamath / lambda-cron Goto Github PK

Currently the lambda function is catching exceptions in the TaskRunner. Do we want to create a way of alerting when this happens? One option is a cloudwatch metric filter on the log group.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-metricfilter.html

http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringLogData.html

When the AWS Lambda function runs it reads all the task definitions (yml files) in the directory (tasks) and evaluate the cron expression for each of them. While the number of task grows performance will be affected negatively.

Proposal:

1. When uploading the tasks to S3 create a file indexing tasks files by cron expression. In this way the AWS Lambda function will read the index file, evaluate different cron expressions only once and will read only the files with the tasks that must be executed.
   Example:

*/30 * * * *:
    - my_task_definition_1.yml
    - my_task_definition_2.yml
0 * * * *:
    - my_task_definiiton_3.yml

For Queue tasks, currently, the tool only allow to indicate the message body to send to the queue. It doesn't allow the possibility to use all options provided by the library.

Proposal:

1. Update task runner to handle all or most useful options provided by boto3 to send a message to a SQS queue.

Currently the only way to disable tasks is to remove from S3 the file that define the task. Be able to disable tasks in the task definition could be useful.

Proposals:

1. Add new attribute in task definition to indicate if the task is enabled or disabled:

   • enabaled: [True|False] (True by default)

2. Replace cron expression with a keyword, ie. disabled

What do you think about raising an exception when an HTTP task returns a non-200 status code? I'm worried that would be easy to ignore an end point returning 500's.

To check the logs of LambdaCron you need to access to AWS CloudWatch using AWS console, CLI or other tool.

I would be useful to add a new command to the CLI to show logs in the terminal in an easy way.

Currently to know the tasks that are deployed in the environment the user needs to list the S3 bucket and browse in the files to see details (ie. cron expresion).

Proposed:
Add new command to list tasks deployed in the S3 bucket and detailed info in a crontab manner:

(cron expresion) (type) (name)

0 * * * *   lambda   run-etl-process-prod
0 9 * * *   queue    my-scheduler-queue

Add new task type to activate AWS Data Pipelines: http://boto3.readthedocs.io/en/latest/reference/services/datapipeline.html#DataPipeline.Client.activate_pipeline

Use a wrapper to allow use all parameters in the function.

Currently LambdaCron allows to set environment options in one file (~/.lambda-cron.yml). Could be good to be able to set environment options in different files (different projects).

Proposal:

1. Be able to include setting files in ~/.lambda-cron.yml. For example:

my-project-a:
    settings: /path/to/my-project-a/lambda-cron.yml
my-project-b:
    settings: /path/to/my-porject-b/lambda-cron.yml

The current version of LambdaCron only allow to send GET and POST requests.

Proposal:

1. Update task runner to handle all (of most of them) HTTP methods available in Requests

Add policies into S3 Bucket to export logs. Following:

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasks.html

The output of the CLI shows the output/results of the commands executed to do the job. It is not user friendly.

Proposal:

1. Hide output from the commands executed by the tool and show meaningful output for the user:
   • Summary of the task running (environment and options)
   • Progress of the process

Currently the validation only run over YAML files (extension .yml).

Proposal
Force all task files to end with ".yml" in the validation.