In-app issue
Incident Report for Help Scout
Postmortem

At Help Scout we use search very heavily to index large amounts of data. When you view the Closed folder, for instance, behind-the-scenes the results are returned from a search index and not directly from the database. We use search in a similar way for dozens of things.

This morning we were testing some improvements that would persist changes to our search indexes faster than it currently works. We tested the changes in our development environment and it went great.

When we put the changes into production, the search cluster locked up because there was too much data hitting it at once. We instantly realized the mistake and it took roughly 16 minutes to get everything back to normal.

Hindsight being 20/20, we understand exactly how this should have been implemented and won't make the same mistake again. Very sorry for the trouble today!

Posted Dec 05, 2014 - 11:49 EST

Resolved
Issue has been resolved and all folders should display properly.
Posted Dec 05, 2014 - 11:04 EST
Update
Public API transactions are also slowing down as a result of this issue and may time out. We're still working on the problem and plan to be back to normal in 5 minutes or less.
Posted Dec 05, 2014 - 11:00 EST
Identified
We're working through an issue causing workflow folders, closed folders and/or spam folders to not display properly. Should be fixed in a few.
Posted Dec 05, 2014 - 10:56 EST