Drupal: Asynchronously send emails with Symfony Mailer Queue

Recently, we built a queue worker to send emails asynchronously, meeting a client's unique requirements to ensure email delivery. There is a lot of movement in the Drupal contribution space to innovate on the mailer. Traditionally, Drupal uses a plain PHP mailer to deliver transactional emails such as sign-up confirmation or password reset instructions. Nowadays, many websites rely on the contribution module Drupal Symfony Mailer to use the framework mailer by Symfony and leverage the flexible setup of mailer policies, transport, and HTML theming with templates. There have also been efforts to integrate Symfony Mailer into Drupal core. One can follow the progress in the meta issue. As of the middle of 2024, the current integration is a proof-of-concept that does not yet support most features, such as email attachments.

Sending emails asynchronously

What needs to be added to the mix are easy-to-implement ways to send emails asynchronously. Today, we announce the first stable release of the new contribution module Symfony Mailer Queue. It processes emails with a queue worker and seamlessly integrates with Drupal Symfony Mailer. In a standard setup, one can use a database queue processed via cron to avoid further infrastructure setup. This solution aims to remain close to the existing Drupal core interfaces and relies entirely on the plugin system in Drupal Symfony Mailer. Still, the implementation is agnostic to the used queue system. Therefore, one may employ other queue workers such as RabbitMQ.

Motivation

When implementing this module, we faced multiple challenges to improve the standard mailer setup. One use case is sending transactional emails after the user submits a complex form. Additional requirements could be one or multiple of the following.

1. Do not send transactional emails during the form submission to enhance the website's responsiveness.
2. Allow complex processing of the submitted data with multiple encapsulated modules interacting with each other.
3. Ensure GDPR compliance where some data is only stored temporarily on the website.
4. Implement a fallback mechanism in case the email delivery fails.

Also, queue processing should be easy to set up through the email policy configuration with Drupal Symfony Mailer. This is achieved using the email adjuster "Queue sending", which can be added to any email policy at Configuration > System > Mailer.

Setting up Ultimate Cron

As mentioned above, when using the database queue and cron to process the queue, one needs to put extra consideration into the cron timings. The usual processing flow is as follows.

1. The user triggers an email, and the "Queue sending" email adjuster in the mailer policy adds an item to the queue.
2. The queue worker claims the item and attempts to send the email. The item is deleted from the queue if the email is sent successfully. Otherwise, the queue item is delayed for a configured amount of time.
3. The expiry of the item is reset during garbage collection after the delay has passed.
4. The queue worker reclaims the item and attempts to send the email until a configured amount of retries is exhausted.
5. The queue worker emits an event that the email delivery failed.

In Ultimate Cron, one can specifically schedule queue processing by activating the option "Override cron queue processing" in Configuration > System > Cron > Cron Settings. In the process above, the queue worker handles steps two, four, and five. The garbage collection in step three would usually be handled during the system cron job. Alternatively, the Symfony Mailer Queue module offers the option to schedule the modules' default cron job that performs the garbage collection for the specific queue. Now, there are only two more things to consider. First, the delay settings in the module configuration at Configuration > System > Mailer > Queue. Second, the way cron triggers on the website. We do not recommend running cron through Automated Cron at the end of a server response. Cron should be externally triggered. Many hosting providers offer the option to do that through, e.g., a crontab configuration.

So, for example, one could use the following configuration:

1. Run cron every minute (triggered externally).
2. Re-queue items after a three-minute delay.
3. Run the Symfony Mailer Queue worker every minute.
4. Run the Symfony Mailer Queue default cron job every minute (garbage collection).

This way, one can ensure that transactional emails are sent shortly after queueing and that retries are scheduled approximately to the configured re-queue delay.

Developer notes

The queue that processes items is required to implement the DelayableQueueInterface to re-queue items. The database queue in Drupal core does implement the required interface. The queue item expiry reset may be performed during cron. In that case, the queue must implement the QueueGarbageCollectionInterface. Again, the database queue in Drupal Core does support garbage collection. Usually, the garbage collection happens during the system cron run. This should be taken into consideration when setting up cron timings. Further, other modules can subscribe to the EmailSendRequeueEvent and EmailSendFailureEvent events to react with logging or further processing.

Importance of stable releases

Motivated by a blog post by Gabe Sullice, we also wanted to tag a stable release for the new contribution module as quickly as possible. That meant adding test coverage for the module's primary processing flow and setting up full linting to ensure code quality with PHP CodeSniffer, CSpell, and PHPStan analysis up to level 8. It is worth mentioning that setting up Gitlab CI is as easy as ever for Drupal contribution modules with the (relatively new) Gitlab Templates.

Similar projects

There has also been a great effort to integrate Symfony Messenger for more advanced use cases. There is an excellent blog series explaining the setup. One may explore the Queue Mail or Mail Entity Queue modules for websites that rely on the traditional PHP mailer.

Conclusion and further reading

In conclusion, the new Symfony Mailer Queue module is a nice addition to the existing Drupal Symfony Mailer ecosystem, enabling asynchronous email processing. It's easy to set up and has been thoroughly tested. We encourage developers to try out the module and share their feedback in the issue queue.

Module: Symfony Mailer Queue

Module: Drupal Symfony Mailer

Module: Ultimate Cron

Drupal Gitlab Templates

Meta issue: Symfony Mailer in Drupal core

Blog series: Symfony Messenger

Blog: Please publish a stable release