Docs Connection Errors

Incident Report for Help Scout

Postmortem

Between 12:37 and 12:55pm EDT today, all public-facing Docs sites were unavailable. The problem ended up being a human error on our end. When we changed DNS records to point to a new load balancer, we discovered that it did not have the correct security settings. The problem was resolved quickly once we updated the security settings.

Our goal is to have several sets of eyes on any infrastructure changes so that mistakes can be avoided. We'll re-evaluate the decision process in this particular instance and do whatever we can to keep this from happening again. We apologize for the brief interruption today.

Posted Oct 28, 2015 - 13:52 EDT

Resolved

We're marking this issue resolved. Docs sites were down for a total of about 18 minutes.

Posted Oct 28, 2015 - 13:24 EDT

Monitoring

Docs sites are back up ... we're monitoring closely and will close this out once we fully understand what happened.

Posted Oct 28, 2015 - 12:56 EDT

Update

We can confirm that all Docs sites are currently down. The Docs API remains up, but DNS is not resolving for public-facing sites. We're working quickly to fix the problem.

Posted Oct 28, 2015 - 12:45 EDT

Investigating

Our automated systems have detected connection problems with public-facing Docs sites and the Ops team has been notified. Stay tuned for further information.

Posted Oct 28, 2015 - 12:37 EDT