Connection Issues
Incident Report for Help Scout

What happened

Starting around 4:10pm EDT on Wednesday, a large number of search events created a high load on our message queueing service. Unfortunately, that sustained high load caused the queuing service to fail. Help Scout services (web app, mobile API, help desk API, and the Docs API) were down for roughly 27 minutes, and services returned to normal around 4:40pm EDT.

What we’re doing about it

We’ve made immediate changes to our infrastructure to keep this particular problem from happening again. Long term, we’re going to separate and expand our messaging queues, which will help distribute high loads more efficiently and redundantly going forward.

Posted 12 days ago. Aug 11, 2017 - 14:15 EDT

Resolved
We're closing this out, as all services have returned to normal. We'll publish a post-mortem in the next 24 hours with a more complete description of what happened and what we're going to do to prevent it in the future.
Posted 14 days ago. Aug 09, 2017 - 17:06 EDT
Monitoring
Page load times have improved, and services are returning to normal. We're still digging for a root cause. We'll continue to monitor until we're in the clear. Inbound and outbound email queues are still delayed. We'll provide a few more updates shortly before closing this out.
Posted 14 days ago. Aug 09, 2017 - 16:49 EDT
Update
The web application is back online, though page loads might be slow as we continue to investigate. Workflows, inbound and outbound email, folder counts, search, and reports will be delayed until we're in the clear.
Posted 14 days ago. Aug 09, 2017 - 16:38 EDT
Update
We're working as quickly as possible to restore services. We'll continue to update here as we make progress.
Posted 14 days ago. Aug 09, 2017 - 16:31 EDT
Investigating
Help Scout is currently unavailable. We're on it.
Posted 14 days ago. Aug 09, 2017 - 16:15 EDT