Connection Issues
Incident Report for Help Scout

What happened

Starting around 4:10pm EDT on Wednesday, a large number of search events created a high load on our message queueing service. Unfortunately, that sustained high load caused the queuing service to fail. Help Scout services (web app, mobile API, help desk API, and the Docs API) were down for roughly 27 minutes, and services returned to normal around 4:40pm EDT.

What we’re doing about it

We’ve made immediate changes to our infrastructure to keep this particular problem from happening again. Long term, we’re going to separate and expand our messaging queues, which will help distribute high loads more efficiently and redundantly going forward.

Posted 11 months ago. Aug 11, 2017 - 14:15 EDT

We're closing this out, as all services have returned to normal. We'll publish a post-mortem in the next 24 hours with a more complete description of what happened and what we're going to do to prevent it in the future.
Posted 11 months ago. Aug 09, 2017 - 17:06 EDT
Page load times have improved, and services are returning to normal. We're still digging for a root cause. We'll continue to monitor until we're in the clear. Inbound and outbound email queues are still delayed. We'll provide a few more updates shortly before closing this out.
Posted 11 months ago. Aug 09, 2017 - 16:49 EDT
The web application is back online, though page loads might be slow as we continue to investigate. Workflows, inbound and outbound email, folder counts, search, and reports will be delayed until we're in the clear.
Posted 11 months ago. Aug 09, 2017 - 16:38 EDT
We're working as quickly as possible to restore services. We'll continue to update here as we make progress.
Posted 11 months ago. Aug 09, 2017 - 16:31 EDT
Help Scout is currently unavailable. We're on it.
Posted 11 months ago. Aug 09, 2017 - 16:15 EDT