Delayed asynchronous invocation using SFN about zappa

[vmiguellima]

@oliviersels I can help with this feature, once you open the PR I can work on it.

Not sure what are the use cases for delayed asynchronous tasks, but one of the most important thing, I think is have the ability to unload jobs/tasks into a queue where there can wait for being picked up.

I like your approach to handle delayed async, I would make a little suggestion:

  "dev": {
    ..
      "async_source": "zappa.async.LambdaAsync",
      "async_policy": "cost-effective",
    ..
    }
}

async_policy allowed values:

• cost-effective, it will decide based on the delayed amount whether to use sqs < 15min delay, or AWS step functions to > 15 min delay
• sqs always use sqs
• step always use step functions

[oliviersels]

Not sure what are the use cases for delayed asynchronous tasks

We use it extensively in our projects for sending reminders to customers. For example:

@task(delay_seconds=60*60*24)  # Delay 24 hours
def send_reminder(user):
    """Check to see if user performed action, otherwise send reminder"""

one of the most important thing, I think is have the ability to unload jobs/tasks into a queue where there can wait for being picked up.

In many cases, SNS already supports this queuing and fanout functionality, including retries, filtering and dead-letter queue handling. I no longer suggest adding SQS as an async source, or maybe only in a different PR, after this one.

I like your approach to handle delayed async, I would make a little suggestion:

 "dev": {
   ..
    "async_source": "zappa.async.LambdaAsync",
    "async_policy": "cost-effective",
   ..
   }
}

async_policy allowed values:

• cost-effective, it will decide based on the delayed amount whether to use sqs < 15min delay, or AWS step functions to > 15 min delay
• sqs always use sqs
• step always use step functions

I'm reluctant to add configuration settings for the async_source as I don't see the need for it if we can contain it in a single class.

{
  "dev": {
    ..
      "async_source": "zappa.async.SfnOrLambdaAsync",
or
      "async_source": "zappa.async.CostEffectiveAsync",
or
      "async_source": "zappa.async.SnsAsync",
or
      "async_source": "myproject.AwesomeAsync",
    ..
    }
}

[jesseduffield]

In many cases, SNS already supports this queuing and fanout functionality, including retries, filtering and dead-letter queue handling. I no longer suggest adding SQS as an async source, or maybe only in a different PR, after this one.

I have no use case for delayed invocations, but I'd like to give my vote for adding SQS as an async source: I'm not an AWS expert but in my mind:

• SQS is for queues where a given message is only consumed by a single consumer
• SNS is for pub-sub

Meaning SQS is a better fit for a background task queue. Although SQS and SNS have lots of overlap, it just seems to me that in the long run SQS is going to be the option that people want to use for background tasks.

[github-actions]

Hi there! Unfortunately, this Issue has not seen any activity for at least 90 days. If the Issue is still relevant to the latest version of Zappa, please comment within the next 10 days if you wish to keep it open. Otherwise, it will be automatically closed.

[github-actions]

Hi there! Unfortunately, this Issue was automatically closed as it had not seen any activity in at least 100 days. If the Issue is still relevant to the latest version of Zappa, please open a new Issue.