Server Error Impacting Performance
Incident Report for Help Scout
Postmortem

What happened

This morning between 7:40 AM and 7:42 AM EST, a small number of customers experienced an internal server error stating that Help Scout services were unavailable. This prevented some customers from accessing various pages in the app, including mailbox and conversations views.

All in all, the service interruption lasted roughly two minutes and was the result of a brief connectivity problem with our web hosting provider. The problem was limited to a single availability zone and only impacted roughly 16% of Help Scout customers.

Here's the official explanation from AWS (Amazon Web Services) during the incident:

"We are investigating a period of elevated error rates for the EC2 APIs and new instance launches in the US-EAST-1 Region. During this period some instances experienced impaired connectivity and some EBS volumes experienced degraded IO performance in a single Availability Zone."

As many SaaS products do, we rely heavily on Amazon Web Services to keep Help Scout online. However, we maintain infrastructure in all East Coast availability zones to minimize the impact if a single zone has a problem. That's precisely what happened today. These types of outages at the provider level are super rare, but when they do happen, there's a chance that a small percentage of customers will experience service interruptions. We're very pleased it was cleared up in a couple of minutes.

Posted Oct 01, 2015 - 15:20 EDT

Resolved
Everything looks good on our end, but we're keeping a close eye on things behind the scenes. We'll have more to share about what happened once we've had some time to really investigate. Very sorry about the hassle this morning!
Posted Oct 01, 2015 - 08:12 EDT
Update
Performance is back to normal and we're no longer seeing errors, though we're still digging for a root cause.
Posted Oct 01, 2015 - 07:56 EDT
Investigating
We're currently investigating some internal server errors that are leading to performance issues in Help Scout. We'll have more to report in just a few minutes.
Posted Oct 01, 2015 - 07:46 EDT