Send bulk email using Amazon SES

After much study of the various ways I could find on the web for bulk delivery tracking using the Amazon Simple Email Service I arrived in a simple and easy-to-control way. And I preferred to use a proprietary solution from the Amazon SDK.

To start amazon does not have the means to manage shipping queues in the SES service, so I chose to use CRON and split shipments into small batches.

I've built an interface to control CRON by PHP , so I can schedule tasks by logging information from a form HTML and processing by PHP . This also allows my CRON table to be left with only one task that will execute a script that will check the tasks of my system.

For this I created a table in the database (I'm using MySQL). By registering the tasks in the database I can easily enable and disable a task, as well as change it as needed.

CREATE TABLE 'cron_task' (
  'id' int(11) NOT NULL AUTO_INCREMENT,
  'description' varchar(250) NOT NULL,
  'minute' varchar(10) NOT NULL,
  'hour' varchar(10) NOT NULL,
  'day' varchar(10) NOT NULL,
  'month' varchar(10) NOT NULL,
  'year' varchar(10) NOT NULL,
  'weekday' varchar(10) NOT NULL,
  'type' varchar(20) NOT NULL,
  'active' tinyint(1) NOT NULL,
  'priority' tinyint(3) NOT NULL,
  'first_execution' timestamp NULL DEFAULT NULL,
  'last_execution' timestamp NULL DEFAULT NULL,
  'created_at' timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  'updated_at' timestamp NULL DEFAULT NULL,
  PRIMARY KEY ('id')
) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8;

I used an interface to standardize my task classes.

interface ICronTask {
    public static function add( ICronTask $task );
    public function activate();
    public function deactivate();
    public function executeTask();
    public function makeLog( $content );
}

Where the add method is used to save a task, activate and deactivate to enable and disable the task, executeTask will contain the task execution context, be it email sending, url request, backup , or whatever else, and the makeLog method to generate custom logs for the execution of the tasks.

I created an abstract class to implement the basic methods for all tasks.

abstract class CronTask implements ICronTask {
    public $id;
    public $description;

    public $minute;
    public $hour;
    public $day;
    public $month;
    public $year;
    public $weekday;

    public $type;
    public $priority;

    public $active;
    public $first_execution;
    public $last_execution;

    public $created_at;
    public $updated_at;

    public $execution_start;
    public $execution_end;

    public function __construct( $id = null ) {
        // caso o $id possua valor, busco informações no banco sobre a tarefa
        // e preencho todos os atributos.
    }

    public final static function add( ICronTask $task ) {
        // Salvo no banco de dados e chamo o método save()
    }

    public final function activate() {
        // Atualizo a tarefa no banco para ativar
    }

    public final function deactivate() {
        // Atualizo a tarefa no banco para desativar
    }

    public final function makeLog( $content ) {
        // Salvo o log da tarefa
    }

    public final function executeTask() {
        // Executo a tarefa
        $this->execution_start = round( microtime(true), 4 );
        $content = $this->execute();
        $this->execution_end = round( microtime(true), 4 );
        $this->makeLog($content);
    }

    public final function isNow() {
        // Faço a verificação da hora de execução, garantindo que deve ser executada
        // no momento em que for chamado.
        return (
            $this->parserTime($this->minute, 'i')   &&
            $this->parserTime($this->hour, 'H')     &&
            $this->parserTime($this->month, 'm')    &&
            $this->parserTime($this->day, 'd')      &&
            $this->parserTime($this->weekday, 'w')
        );
    }

    private function parserTime( $value, $element ) {
        // Obtem o tem atual
    $time = date( $element );

    // Verifica se o valor é igual à "*" representando toda momento.
    if( $value == '*' ) {
        return true;
    }

    // Separa os conjuntos de tempos separados por vírgula
    $groups = explode( ',', $value );
    foreach ( $groups as $part ) {
        // Verifica se é um intervalo composto. Ex: "*/5" ou "20-40/2"
        // Se é um intervalo compost, deverá retornar true se o valor atual
        // estiver dentro do intervalo definido antes da barra, e na frequência
        // definida após a barra.
        if( strpos( $part, '/' ) ) {
            $groupsInterval = explode( '/', $part );
            // Verificando a frequência
            $frequency = $time % $groupsInterval[1] == 0;

            // Verificando o intervalo
            $interval = explode( '-', $groupsInterval[0] );
            $intervalResult = false;
            if( $interval[0] == '*' ) {
                $intervalResult = true;
            } else {
                $intervalResult = ( $time >= $interval[0] && $time <= $interval[1] );
            }
            return $frequency && $intervalResult;
        }

        // Verifica se é um intervalo simples. Ex: "10-50"
        // Se é um intervalo, deverá retornar true se o valor atual estiver
        // dentro desse intervalo.
        if( strpos( $part, '-' ) ) {
            $interval = explode( '-', $part );
            return $time >= $interval[0] && $time <= $interval[1];
        }

        // Se for um número simples verifica se é o tempo certo
        if( $time == $part ) {
            return true;
        }
    }
    return false;
    }

    abstract protected function execute();
    abstract protected function save();
}

And an example implementation of a task class would be

class CronTaskTest extends CronTask {
    public $type = 'Test';
    public $priority = 0;

    protected function execute() {
        return 'Tarefa executada com sucesso';
    }

    protected function save() {
        return true;
    }
}

The main class that will be executed at all times verifying the tasks was implemented as follows:

class Cron {
    public static function execute() {
        $tasks = self::getTasks();
        foreach ( $tasks as $task ) {
            if( $task->isNow() ) {
                $task->executeTask();
            }
        }
    }

    public static function getTasks() {
        try {
            $tasks = // Busco todas as tarefas ativas ordenadas por prioridade DESC;

            $return = array();
            foreach ( $tasks as $record ) {
                $taskName = 'CronTask' . $record['type'];
                require_once __DIR__ . '/tasks/' . $taskName . '.php';
                $return[] = new $taskName( $record['id'] );
            }
        } catch ( PDOException $exception ) {
            die( $exception->getMessage() );
        }
        return $return;
    }
}

I create a PHP file to execute the tasks by calling the Cron::execute() method. And working on CRON

# crontab -e
* * * * * /usr/local/bin/php /var/www/projeto/meu-script.php

So I created a class called CronTaskMailing similar to the example above with the attributes themselves

public $name;
public $subject;
public $body;
public $alternativeBody;
public $startSend;
public $notifyTo;
public $keywords = array();
public $addresses = array();
public $sended = array();

public $startedAt;
public $completedAt;

These attributes I save in a json . For this I implemented logic in the save method that is called soon when the task is added. Also implement another method to load the information of json saved to the attributes of the class.

I've implemented a method to handle the entire message that will be passed to the Amazon SDK.

private function makeMessage() {
    $msg = array();
    $msg['Source'] = "[email protected]";

    $msg['Message']['Subject']['Data'] = $this->subject;
    $msg['Message']['Subject']['Charset'] = "UTF-8";

    $msg['Message']['Body']['Text']['Data'] = $this->alternativeBody;
    $msg['Message']['Body']['Text']['Charset'] = "UTF-8";

    $msg['Message']['Body']['Html']['Data'] = $this->body;
    $msg['Message']['Body']['Html']['Charset'] = "UTF-8";

    return $msg;
}

Define a maximum average usage of SES for mailing, preventing other services that also use SES from being unable to use. This quota is set in percentage to facilitate the accounts.

const MAX_QUOTA_USAGE = 0.8;

And the method of executing the task was as follows:

protected function execute() {
    $log = '';
    try {
        $this->loadMailingFile( $this->id );
        $ses = SesClient::factory( array(
            'key' => 'ACCESS_KEY',
            'secret' => 'API_SECRET',
            'region' => 'REGION'
        ) );

        if( $this->startedAt ) {
            $this->startedAt = time();
        }

        $msg = $this->makeMessage();

        // Obtenho a minha cota e verifico se está dentro da média máxima definida.
        $quota = $ses->getSendQuota();
        $maxQuota = $quota['Max24HourSend'] * self::MAX_QUOTA_USAGE;

        if( $maxQuota < $quota['SentLast24Hours'] ) {
            $log .= 'Cota máxima para mailing em 24 horas excedida.';
        } else {
            // Calculo o delay entre uma mensagem e outra para não 
            // estourar o número de envios por segundo.
            $rate = $quota['MaxSendRate'];
            $delay = $quota['MaxSendRate'] / 1000000;
            // Calculo o tamanho do lote para que a execução dure no máximo um minuto.
            $maxSendBatch = $rate * 60;

            $count = 0;
            foreach ( $this->addresses as $key => $email ) {
                $msg['Destination']['ToAddresses'] = array( $email );
                $response = $ses->sendEmail( $msg );
                $messageId = $response->get( 'MessageId' );
                $log .= 'Mensagem enviada: ' . $messageId . PHP_EOL;

                $this->sended[] = array(
                    'email' => $email,
                    'message_id' => $messageId,
                    'sended_at' => time()
                );
                unset( $this->addresses[$key] );

                $count++;
                if( $count == $maxSendBatch ) {
                    break;
                }
                usleep($delay);
            }
            $log .= '-----' . PHP_EOL;
            $log .= 'Total de emails enviados: ' . count( $this->sended ) . PHP_EOL;
            $log .= 'Total de emails que faltam: ' . count( $this->addresses ) . PHP_EOL;
            if( count( $this->addresses ) == 0 ) {
                $this->completedAt = time();
                $this->deactivate();
                // Método para notificar por email quando o envio terminar.
                $this->sendNotificationToMailingCompleted();
                $log .= 'Envio de mailing concluído';
            }
        }
        $this->save();
    } catch (Exception $ex) {
        $log .= $ex->getMessage() . PHP_EOL;
    }
    return $log;
}

The quota increases gradually according to the good use of the service, so I worked out the code above in order to identify this increase and work with the highest possible shipping rate.

In Amazon's initial settings (10,000 emails in 24 hours with 5 emails per second) would be 300 emails per minute, meaning it would take 33 minutes to send 10,000 emails on average. It may take a few more hours using the way I quote here, depending on the limit set at constant MAX_QUOTA_USAGE .

In addition, Amazon has strict policies for sending email, and having a clean email list, that is, without many invalid emails is essential. To resolve this problem, I implemented another task that performs verifying all emails every 15 days, and storing information indicating whether or not the email is valid.

The Amazon SDK has method that enables this verification (verifyEmailIdentity) . But this method should be called with the interval of at least 1 second.