Connection Issues

Incident Report for Help Scout

Postmortem

Starting at 9:43am EDT today, Help Scout experienced degraded performance until roughly 10:12am EDT. Page loads took an average of 10 seconds to load, instead of the average we typically see, around 2 seconds.

This was a repeat of an issue we saw on June 6, but this time we have been able to identify the root cause, which leads back to our memcache instances.

Our Ops team is working hard on a fix to prevent this issue from happening again, which will be deployed by the end of the week. In the meantime, we'll keep a very close eye on memcache and should be able to clear up any recurrences more quickly.

So sorry for the less than stellar performance between this and the last issue on June 6. We'll be excited to patch this up once and for all.

Posted Jun 15, 2016 - 11:48 EDT

Resolved

This morning around 9:45am EDT, we started seeing slow page load times across the web app. We’re still investigating the cause of today's sluggishness, and will post a follow-up with more information as soon as possible. At roughly 10:15am EDT, all services returned to normal.

Posted Jun 15, 2016 - 10:33 EDT

Monitoring

Page load times are returning to normal across the app. Inbound and outbound email is still delayed, but we're starting to clear the queues now. Workflows and folder counts are catching-up as well. We'll continue to monitor performance for a bit before closing this out.

Posted Jun 15, 2016 - 10:16 EDT

Update

We're still investigating slow page loads across the web app. We're working on getting services back to a stable point. Inbound and outbound email will be delayed until we can get this sorted. Additionally, folder views/counts, as well as workflows, will be slow to update until we're in the clear.

Posted Jun 15, 2016 - 10:00 EDT

Investigating

We're investigating reports of slow page load times and connection timeouts across the app. We'll continue to update here as we have more information.

Posted Jun 15, 2016 - 09:48 EDT