What happened

Starting around 4:10pm EDT on Wednesday, a large number of search events created a high load on our message queueing service. Unfortunately, that sustained high load caused the queuing service to fail. Help Scout services (web app, mobile API, help desk API, and the Docs API) were down for roughly 27 minutes, and services returned to normal around 4:40pm EDT.

What we’re doing about it

We’ve made immediate changes to our infrastructure to keep this particular problem from happening again. Long term, we’re going to separate and expand our messaging queues, which will help distribute high loads more efficiently and redundantly going forward.

Posted Aug 11, 2017 - 14:15 EDT

Resolved

We're closing this out, as all services have returned to normal. We'll publish a post-mortem in the next 24 hours with a more complete description of what happened and what we're going to do to prevent it in the future.

Posted Aug 09, 2017 - 17:06 EDT

Monitoring

Page load times have improved, and services are returning to normal. We're still digging for a root cause. We'll continue to monitor until we're in the clear. Inbound and outbound email queues are still delayed. We'll provide a few more updates shortly before closing this out.

Posted Aug 09, 2017 - 16:49 EDT

Update

The web application is back online, though page loads might be slow as we continue to investigate. Workflows, inbound and outbound email, folder counts, search, and reports will be delayed until we're in the clear.

Posted Aug 09, 2017 - 16:38 EDT

Update

We're working as quickly as possible to restore services. We'll continue to update here as we make progress.

Posted Aug 09, 2017 - 16:31 EDT

Investigating

Help Scout is currently unavailable. We're on it.

Posted Aug 09, 2017 - 16:15 EDT