Between 2:30pm and 3:00pm EST today, Help Scout was experiencing elevated page load times, and roughly 5-10 minutes of downtime.

The cause was an abnormally high load of activity on the queues that process folder counts, Workflows and outgoing email. In an effort to manage the additional load, we increased the number of consumers processing the queues and it had a negative impact on performance in the web application. We backed off the additional consumers and restarted the web servers to get things back to normal.

We don't expect any further issues today. However, the queues are still backed up and it will take a couple of hours (up to 5:35pm EST) to get folder counts, workflows and outgoing emails up-to-date.

There's a lot we can take away from this experience, and there are most definitely some architectural changes that need to be made in order to provide you with a more reliable experience under high load. We're on it!