bahrmichael / aws-scheduler

A scheduler for large amounts of time precise events

New Point In Time Scheduler

I've released a new scheduler service: Check out the launch blog post.

Old aws-scheduler

!! This project is no longer actively maintained. !!

Warning: I've rebuilt some parts of the system. If you followed the setup below before 01.06.20202, you have to recreate the database. For zero downtime I suggest you deploy the new version, switch over the input topic and then let the old version run dry before switching it off.

This project provides a solution to schedule large amounts of point in time events with a great time precision. See the Performance section for more details.

The two interfaces are two SNS topics. One for input and one for output. You send a scheduling payload to the input topic and at the specified datetime you will receive your data at the output topic. See the Usage section on how to attach your functions.

Usage

A quick start project is available at aws-scheduler-testing.

Setup

First of all you need an output topic that we can publish events to once the scheduled datetime arrives. To do this run python setup/init_output_topic.py <stage>. This will create a topic called scheduler-output-<stage> and grant our account the right to publish messages. You can see the added policy below.

{
  "Sid": "scheduler-output-<stage>-publish-access",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:sts::256608350746:assumed-role/aws-scheduler-prod-us-east-1-lambdaRole/aws-scheduler-prod-emitter"
  },
  "Action": "SNS:Publish",
  "Resource": "arn:aws:sns:us-east-1:256608350746:scheduler-output-<stage>"
}

If you don't want to grant our emitter the right to publish, you can pass your own emitter role as a third argument.

Creating topic scheduler-output-<stage>
Created topic scheduler-output-<stage> with arn arn:aws:sns:us-east-1:<your-account-id>:scheduler-output-<stage>
Granting publish rights to scheduler-output-<stage> for role arn:aws:sts::256608350746:assumed-role/aws-scheduler-prod-us-east-1-lambdaRole/aws-scheduler-prod-emitter
Done

Write down the ARN of your output topic as you will need it for the input events.

Rerun this process with the command python setup/init_failure_topic.py <stage> to create a topic where the service can publish errors.

Input

To schedule a trigger you have to publish an event which follows the structure below to the ARN of the input topic. You can find the ARN of our service in the SAAS Offer section.

{
  "date": "utc timestamp following ISO 8601",
  "target": "arn of your sns output topic",
  "user": "some way we can get in touch with you",
  "payload": "any string payload",
  "failure_topic": "arn of an sns topic where the service can publish errors"
}

All fields except failure_topic are mandatory. Please make sure that the payload can be utf-8 encoded. If you submit an event that does not follow the spec, it will published to the failure_topic.

SNS messages must be strings. First string encode the json structure and then publish it to the input topic.

# Python example
import json
import boto3

client = boto3.client('sns')

event = {
          "date": "2019-07-27T12:20:24.919071",
          "target": "arn:aws:sns:us-east-1:256608350746:scheduler-output-prod",
          "user": "Twitter @michabahr",
          "payload": "46607451-3e67-49bc-972b-425c150c5456"
        }

input_topic = 'arn:aws:sns:us-east-1:256608350746:scheduler-input-prod'
client.publish(TopicArn=input_topic, Message=json.dumps(event))

So far there is no batch publishing available for SNS. Make sure the event stays within the 256KB limit of SNS. We recommend that you only submit IDs and don't transfer any real data to the service.

Output

Once the datetime specified in the payload is reached, the service will publish the content of the event field to your output topic. You can attach an AWS Lambda function to your topic to process the events.

Deploy it yourself

This section explains how you can deploy the service yourself. Once set up use it like shown above.

The following picture shows you the structure of the service.

Prerequisites

You must have the following tools installed:

• serverless framework 1.48.3 or later
• node
• npm
• python3
• pip

Run setup/init_table.py <stage> to setup the database. Replace <stage> with the stage of your application.

Run setup/init_input_topic.py <stage> [public]. Replace <stage> with the stage of your application. You may append the parameter public to grant public publish rights.

Run setup/init_queue.py <stage>. Replace <stage> with the stage of your application.

Deploy

1. Navigate into the project folder
2. With a tooling of your choice create and activate a venv
3. pip install -r requirements.txt
4. npm i serverless-python-requirements
5. sls deploy

Wait for the deployment to finish. Test the service by first attaching a function to the output topic and then send a few events to the input topic.

You can disable the user check on the input event for your own deployment with the ENFORCE_USER environment variable.

Performance

We ran tests by sending events that included the scheduled timestamp in the payload. Once received we compared those timestamps with the current time.

In our results all events arrive within one second with a clear trend to stay well below 100ms.

The following chart show the amount of events received on the y axis and the distribution by delay on the x axis.

Scalability

Based on our tests from the Performance section, we are confident that this stack can handle 100.000.000 events per month and might scale up to 500.000.000 events per month.

These numbers are not confirmed though, as that volume incurs significant AWS costs.

Limitations

• Events may arrive more than once at the output topic.
• This approach costs more than using DynamoDB's TTL attribute. If delays of 30 minutes to 48 hours are acceptable for you, then check out this article.

Contributions

Contributions are welcome, both issues and code. Get in touch at twitter @michabahr or create an issue.

Example payloads

Output topic

{
  "EventSource": "aws:sns", 
  "EventVersion": "1.0", 
  "EventSubscriptionArn": "...", 
  "Sns": {
    "Type": "Notification", 
    "MessageId": "eccc1539-0867-5c6d-8b53-408f5b91b578", 
    "TopicArn": "arn:aws:sns:us-east-1:256608350746:scheduler-output-prod", 
    "Subject": null, 
    "Message": "my-message", 
    "Timestamp": "2019-07-27T13:29:26.405Z", 
    "SignatureVersion": "1", 
    "Signature": "...", 
    "SigningCertUrl": "...", 
    "UnsubscribeUrl": "...", 
    "MessageAttributes": {}
  }
}

TODOs

• adjust pictures to show failure queue
• limitation of message size (10kb), also explain why
• secure the PoC with test
• add a safe guard that pulls messages from dead letter queues back into the circuit
• handling for messages that can't be utf-8 encoded

aws-scheduler's Issues

Question: Any Delay limit?

Is there any delay limit?
Can I use up-to one year?
I can see it using time.sleep(delay). So it's a blocking code. Will that really be feasible?
How may I be billed in the AWS Lambda for a one year sleep?

Deployment error - ConsumerLambdaFunction validation error

Hey @bahrmichael ,

Thanks for sharing this piece of code. I am trying to deploy but it's giving me attached error.

I tried everything within my knowledge as i am from C# background, but nothing is working.

Thanks,

Error "A provided account ID is not valid" when trying init output topic

When I try to run init_output_topic.py I get error

$ python3 ./setup/init_output_topic.py dev public
Creating topic scheduler-output-v2-dev
Created topic scheduler-output-v2-dev with arn arn:aws:sns:us-east-1:829736848236:scheduler-output-v2-dev
Granting publish rights to scheduler-output-v2-dev topic for role public
Traceback (most recent call last):
  File "./setup/init_output_topic.py", line 28, in <module>
    permission_response = client.add_permission(
  File "/home/apollon/.local/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/apollon/.local/lib/python3.8/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameter) when calling the AddPermission operation: Invalid parameter: A provided account ID is not valid

Question: why set TTL to 10 mins AFTER scheduled time?

I'm looking at this line of code and it's not making sense to me. Why set TTL to 10 mins AFTER the scheduled time? Considering that DynamoDB TTL doesn't delete item right away (when it's not a busy table, this can be anything between 10-30 mins after an item is expired) so why would you add even more delay to that process?