Partial outage

Incident Report for VisualVault

Postmortem

Root Cause

Upon investigation, our team identified that the root cause of the disruption was a failure within one of the nodes in the backend database cluster. This caused a temporary loss of connectivity to the affected systems. We immediately initiated our internal protocols to mitigate the impact.

Resolution

Our engineering team quickly pinpointed the issue and took the necessary steps to restore full service. By 21:59 MST the issue was fully resolved and all impacted systems were brought back online. Since then, the systems have been operating as expected.

Steps Taken

  • Immediate identification of the affected systems and root cause.
  • Activation of our internal failover processes to minimize impact.
  • Ongoing monitoring to ensure all systems remain stable.
  • A thorough post-incident review to prevent recurrence.

What We're Doing to Prevent This in the Future

We are committed to preventing similar incidents in the future. To this end, we are:

  • Conducting a comprehensive review of our database infrastructure to improve reliability.
  • Implementing additional monitoring tools to detect potential issues earlier.
  • Exploring further redundancies in our backend systems to ensure higher availability.

Conclusion

We take reliability seriously and deeply regret the disruption you experienced. We value your trust and want to reassure you that we are actively working to enhance our infrastructure to better serve you moving forward. If you have any further questions or concerns, please don’t hesitate to reach out to

Posted Apr 01, 2025 - 22:02 MST

Resolved

The operations team has resolved this issue.
Posted Apr 01, 2025 - 21:59 MST

Investigating

We are investigating an outage affecting these VisualVault instance URLs:
https://orc.grmims.com
https://na4.visualvault.com
Posted Apr 01, 2025 - 21:21 MST
This incident affected: VisualVault - GRMIMS (VisualVault Web UI, REST API, Document Viewing, eScan) and VisualVault - North America 4 (NA4) (VisualVault Web UI, REST API, Document Viewer).