What happened

Starting around 6:13pm EDT on Thursday, an unusually high spike of messages caused our message queueing services to fail. Help Scout services (web app, mobile API, and help desk API ) were down for roughly 18 minutes, and services returned to normal around 6:31pm EDT.

What we're doing about it

We're working to separate our message queue cluster to spread any unusual spikes like this across several clusters. Going forward, we're also adding new alerts, as well as additional error logging around our internal APIs.

Posted Sep 01, 2017 - 21:34 EDT

Resolved

We're closing this out, as all services have returned to normal. We'll continue to monitor over the next few hours. We're still digging for a root cause, and will publish a post-mortem in the next 24 hours with a more complete description of what happened.

Posted Aug 31, 2017 - 18:40 EDT

Update

The web application, help desk API, and mobile apps are back online, though page loads might be slow as we continue to investigate. Workflows, inbound and outbound email, folder counts, search, and reports will be delayed until we're in the clear.

Posted Aug 31, 2017 - 18:31 EDT

Update

We're working as quickly as possible to restore services. We'll continue to update here as we make progress.

Posted Aug 31, 2017 - 18:23 EDT

Investigating

Help Scout is currently unavailable. We're investigating.

Posted Aug 31, 2017 - 18:16 EDT