Starting around 8:00 AM UTC, an automated update did not run correctly on one of our search clusters, which caused the service to fail. As a result, folders (Closed, Workflow folders) and search results were not showing updates as expected. We didn't catch this soon enough, as our usual alerts did not go off. As of this update, all search services are stable. New events and conversation updates are processing as they should be.
No data was lost while the search cluster was out of service, but there's a large backlog of search events that we're processing now. Over the next few hours, you'll start to see your Closed folder, search results, and reporting drill downs populate with conversation updates from earlier in the day. Additionally, workflows that did not fire right away due to the search delay will fire soon. Tags are the final piece of the puzzle. If you visit the Tags management page, and you're not seeing newly created tags, those tags will populate once the data backlog has cleared.
Again, to be clear, no data was lost during this event. All customer emails were sent and delivered, except for delayed Workflow emails, as mentioned above. There's nothing you need to do in Help Scout (or outside of Help Scout) during the re-queue process. You can keep working as usual. We're very sorry for the trouble this caused today, and we're sorry for dropping the ball on communicating all of this as it was happening. We'll publish a full post-mortem once we've had time to recover and take closer look at the root problem.