Some outgoing emails delayed

Incident Report for Help Scout

Postmortem

Yesterday afternoon around 3pm EST we discovered an issue with one of our mail servers. All internal monitoring said the server was fine and our outbound queues were clear, but this server was returning an error when attempting to send email.

All emails sent to the impacted mail server for processing failed and were backed up to another location. We have three main outgoing email servers, so roughly 2/3 of outgoing emails were sent immediately, without delay. The other 1/3 was backed up and queued for delivery via our other servers by 5pm EST.

No data was lost in this process and all delayed emails were processed by 8pm EST. Emails include notifications to customers and replies to customers.

We are able to prevent this in the future by adding more monitoring to the database table that stores outbound email errors. That way we'll be notified immediately if this issue ever comes up again.

Posted Jan 05, 2015 - 08:58 EST

Resolved

All outgoing email is back to normal!

Posted Jan 04, 2015 - 20:05 EST

Update

The outbound email delays today only impacted roughly 1/3 of emails queued for delivery, and they will all be sent by 8pm EST tonight. The other 2/3 were delivered immediately as normal. No data was lost in the process, but the unlucky 1/3 were delayed. We'll post another update when the outbound queue is clear and start on a postmortem.

Posted Jan 04, 2015 - 18:30 EST

Monitoring

Our outbound email queue should be all clear within 2 hours. We'll keep an eye on this and won't mark this issue as resolved until it's up-to-date.

Posted Jan 04, 2015 - 18:06 EST

Update

We're still down an email server and have a queue of roughly 40k emails right now, but are working through them quickly.

Posted Jan 04, 2015 - 17:50 EST

Identified

We've identified an issue with one of our email servers that's delaying outbound sending. This impacts emails to customers and email notifications. We'll post more information as we have it ...

Posted Jan 04, 2015 - 17:11 EST