Support for Boto3 about django-ses

[Sammaye]

I know this is old but for those who just want the backend I actually converted the main part of this script to boto3:

from django.conf import settings
from django.core.mail.backends.base import BaseEmailBackend

from boto3.session import Session
import botocore

from datetime import datetime, timedelta
from time import sleep

settings.AWS_SES_AUTO_THROTTLE = getattr(settings, 'AWS_SES_AUTO_THROTTLE', 0.5)

settings.DKIM_DOMAIN = getattr(settings, "DKIM_DOMAIN", None)
settings.DKIM_PRIVATE_KEY = getattr(settings, 'DKIM_PRIVATE_KEY', None)
settings.DKIM_SELECTOR = getattr(settings, 'DKIM_SELECTOR', 'ses')
settings.DKIM_HEADERS = getattr(settings, 'DKIM_HEADERS',
            ('From', 'To', 'Cc', 'Subject'))

TIME_ZONE = settings.TIME_ZONE

# These would be nice to make class-level variables, but the backend is
# re-created for each outgoing email/batch.
# recent_send_times also is not going to work quite right if there are multiple
# email backends with different rate limits returned by SES, but that seems
# like it would be rare.
cached_rate_limits = {}
recent_send_times = []


def dkim_sign(message, dkim_domain=None, dkim_key=None, dkim_selector=None, dkim_headers=None):
    """Return signed email message if dkim package and settings are available."""
    try:
        import dkim
    except ImportError:
        pass
    else:
        if dkim_domain and dkim_key:
            sig = dkim.sign(message,
                            dkim_selector,
                            dkim_domain,
                            dkim_key,
                            include_headers=dkim_headers)
            message = sig + message
    return message


class SESBackend(BaseEmailBackend):
    """A Django Email backend that uses Amazon's Simple Email Service.
    """

    def __init__(self, fail_silently=False, **kwargs):

        super(SESBackend, self).__init__(fail_silently=fail_silently, **kwargs)

        self.session = Session(
            aws_access_key_id = settings.AWS_ACCESS_KEY_ID,
            aws_secret_access_key = settings.AWS_SECRET_ACCESS_KEY,
            region_name = settings.AWS_REGION_NAME
        )

        self._throttle = settings.AWS_SES_AUTO_THROTTLE

        self.dkim_domain = settings.DKIM_DOMAIN
        self.dkim_key = settings.DKIM_PRIVATE_KEY
        self.dkim_selector = settings.DKIM_SELECTOR
        self.dkim_headers = settings.DKIM_HEADERS

        self.connection = None

    def open(self):
        """Create a connection to the AWS API server. This can be reused for
        sending multiple emails.
        """
        if self.connection:
            return False

        try:
            self.connection = self.session.client('ses')
        except:
            if not self.fail_silently:
                raise

    def close(self):
        """Close any open HTTP connections to the API server.
        """
        try:
            #self.connection.close()
            self.connection = None
        except:
            if not self.fail_silently:
                raise
    def send_messages(self, email_messages):
        """Sends one or more EmailMessage objects and returns the number of
        email messages sent.
        """
        if not email_messages:
            return

        new_conn_created = self.open()
        if not self.connection:
            # Failed silently
            return

        num_sent = 0
        for message in email_messages:
            # Automatic throttling. Assumes that this is the only SES client
            # currently operating. The AWS_SES_AUTO_THROTTLE setting is a
            # factor to apply to the rate limit, with a default of 0.5 to stay
            # well below the actual SES throttle.
            # Set the setting to 0 or None to disable throttling.
            if self._throttle:
                global recent_send_times

                now = datetime.now()

                # Get and cache the current SES max-per-second rate limit
                # returned by the SES API.
                rate_limit = self.get_rate_limit()

                # Prune from recent_send_times anything more than a few seconds
                # ago. Even though SES reports a maximum per-second, the way
                # they enforce the limit may not be on a one-second window.
                # To be safe, we use a two-second window (but allow 2 times the
                # rate limit) and then also have a default rate limit factor of
                # 0.5 so that we really limit the one-second amount in two
                # seconds.
                window = 2.0  # seconds
                window_start = now - timedelta(seconds=window)
                new_send_times = []
                for time in recent_send_times:
                    if time > window_start:
                        new_send_times.append(time)
                recent_send_times = new_send_times

                # If the number of recent send times in the last 1/_throttle
                # seconds exceeds the rate limit, add a delay.
                # Since I'm not sure how Amazon determines at exactly what
                # point to throttle, better be safe than sorry and let in, say,
                # half of the allowed rate.
                if len(new_send_times) > rate_limit * window * self._throttle:
                    # Sleep the remainder of the window period.
                    delta = now - new_send_times[0]
                    total_seconds = (delta.microseconds + (delta.seconds +
                                     delta.days * 24 * 3600) * 10**6) / 10**6
                    delay = window - total_seconds
                    if delay > 0:
                        sleep(delay)

                recent_send_times.append(now)
                # end of throttling
            try:
                response = self.connection.send_raw_email(
                    Source = settings.DEFAULT_FROM_EMAIL,
                    Destinations = message.recipients(),
                    RawMessage = { 'Data': dkim_sign(
                        message.message().as_string(),
                        dkim_key=self.dkim_key,
                        dkim_domain=self.dkim_domain,
                        dkim_selector=self.dkim_selector,
                        dkim_headers=self.dkim_headers
                    )}
                )

                message.extra_headers['status'] = response['ResponseMetadata']['HTTPStatusCode']
                message.extra_headers['message_id'] = response['MessageId']
                message.extra_headers['request_id'] = response['ResponseMetadata']['RequestId']
                num_sent += 1
            except botocore.exceptions.ClientError as error:
                # Store failure information so to post process it if required
                print(error)
                error_keys = ['status', 'reason', 'body', 'request_id',
                              'error_code', 'error_message']
                for key in error_keys:
                    message.extra_headers[key] = getattr(error, key, None)
                if not self.fail_silently:
                    raise

        if new_conn_created:
            self.close()

        return num_sent
    def get_rate_limit(self):
        if settings.AWS_ACCESS_KEY_ID in cached_rate_limits:
            return cached_rate_limits[settings.AWS_ACCESS_KEY_ID]

        new_conn_created = self.open()
        if not self.connection:
            raise Exception(
                "No connection is available to check current SES rate limit.")
        try:
            quota_dict = self.connection.get_send_quota()
            print(quota_dict)
            max_per_second = quota_dict['MaxSendRate']
            ret = float(max_per_second)
            cached_rate_limits[settings.AWS_ACCESS_KEY_ID] = ret
            return ret
        finally:
            if new_conn_created:
                self.close()

[pcraciunoiu]

@fission6 I haven't tried it. You are welcome to try installing it as a dependency and fix up any issues you see. Submit that as a Pull Request and we can work together to get it working. I'll help with code reviews and stuff, but I'm a just too swamped to do it myself right now. The beauty of open source is that you can do it, and it might only take you an hour :)

[pcraciunoiu]

Thanks for the update @StevenMapes -- I'm not actively using this project so I haven't spent time on it. But if you wish to submit a PR with details, I can review.

[pcraciunoiu]

Good point @pjxiao. In that case I'm OK dropping boto2 support. We can just keep the current master on a py2 branch in case of hotfixes and move forward with the world.

If either of you take this on, it'd be great if you can update the README to specify python 2 and boto 2 are both deprecated starting with version 1, and we'll do a major bump once the PR(s) land(s).

[GitRon]

I started a PR for this topic, though I'd need some help. More details are here: #183

[bblanchon]

Boto3 is supported now, you can safely close this issue.

[zheli]

It would be great if django-ses merge can this change...

[pcraciunoiu]

If we can make this a Pull Request, I can make time to review and merge it in...

@zheli @Sammaye @bwebsterdv

[harkamals]

Brilliant !
I just had to add proxy to get it to work, thanks much !!

import os; os.environ["https_proxy"] = "https://myproxyserver:8080"

[davideghz]

is boto3 supported now?

[fission6]

would love an answer to this

is boto3 supported now?

[StevenMapes]

@Sammaye works but its best to tweak it a little first.

I replaced the print() in get_rate_limit with a call to a logger and recommend using a connection to sts to determine the access key that is being used and move away from using inline access credentials to support named profiles and, more over, environmental/IAM role based credentials.

    def get_rate_limit(self):
        access_key_id = get_boto3_client("sts").get_caller_identity().get('UserId')
        if access_key_id in cached_rate_limits:
            return cached_rate_limits[access_key_id]

        new_conn_created = self.open()
        if not self.connection:
            raise Exception(
                "No connection is available to check current SES rate limit.")
        try:
            quota_dict = self.connection.get_send_quota()
            logger.debug(quota_dict)
            max_per_second = quota_dict['MaxSendRate']
            ret = float(max_per_second)
            cached_rate_limits[access_key_id] = ret
            return ret
        finally:
            if new_conn_created:
                self.close()

I've added support for that my removing the session setup and changing the connection setup in open to call a factory function I have that supports connecting via all three methods. This way it will work on an EC2 server using the IAM profile of the machine which is better for security and means less setup within your settings, locally and outside of AWS you can use the .aws/config files to create a profile and use named profiles, but it also supports legacy setups using direct credentials.

I use this and a similar wrapper to setup client and resource connections via boto3 all the time.

def get_boto3_client(service):
    """Return the Boto3 client.

    Supports all three methods of connection

    1. Uses a PROFILE stored on the machine within the .aws folder. Preferred for LOCAL development
    2. Used named settings. This is the most insecure method but is used older applications - LEGACY
    3. Uses environmental settings such as the IAM Profile of the running EC2 instance. PREFERRED METHOD

    :param str service:
    :return:
    :rtype obj:
    """
    if hasattr(settings, 'AWS_IAM_PROFILE') and settings.AWS_IAM_PROFILE:
        # Used a named profile where the credentials are stored within the .aws folder. Preferred Non-AWS method
        session = boto3.Session(profile_name=settings.AWS_IAM_PROFILE)
        return session.client(service)
    elif hasattr(settings, 'AWS_ACCESS_KEY_ID') and hasattr(settings, 'AWS_SECRET_ACCESS_KEY') and \
            hasattr(settings, 'AWS_REGION_NAME') and settings.AWS_ACCESS_KEY_ID and settings.AWS_SECRET_ACCESS_KEY \
            and settings.AWS_REGION_NAME:
        # Worst method, allows direct credentials to be used
            session = boto3.Session(
                aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
                aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
                region_name=settings.AWS_REGION_NAME
            )
            return session.client(service)
    else:
        # Uses credentials at the environmental level. Uses IAM roles associated to the AWS resource. Preferred method
        return boto3.client(service, region_name=settings.AWS_S3_REGION_NAME)

This is a great project but it's a shame that its not really maintained these days and using boto2 means it's a non-starter for me

[StevenMapes]

Thanks for the update @StevenMapes -- I'm not actively using this project so I haven't spent time on it. But if you wish to submit a PR with details, I can review.

I will, I've cloned it with the intention of running through everything to port it to boto3 only and add in some additional functionality I need for a couple of projects over the next month only I may end up not using it in the meantime and coming back to this afterwards to retrofit it back it.

How against dropping boto2 support are you?

[pcraciunoiu]

Based on their docs, looks like boto2 is still supported. So if it's not a lot of extra work, it seems nice to keep it for now, so we don't force any current users to upgrade before they're ready. I haven't used boto2 in so long, that I wanted to say yes... but it's everyone using this project is on boto2.

You could throw a warning somewhere and add to the README that the plan is to deprecate it?

[pjxiao]

Boto2 seems to have a py3 compatible problem and this problems haven't been repaired for several years although there're pull request which repare this.
For instance, PR boto/boto#3699, which prevents me from calling the SES API using proxy_ssl, was opened in 2017 and is stilled opened today.