On Jan 25 at approximately 0530 MT a failing maintenance and backup process began causing back-pressure up the replication line of Teem databases. This in turn caused normal and overflow disk space to be filled and eventually led to a primary database becoming unavailable and the service disruption that followed. The failing process was identified and corrected, the blockage released and the systems were brought back online at 1020 MT.
Posted Feb 07, 2020 - 17:05 MST
Resolved
On Jan 25 at approximately 0530 MT a failing maintenance and backup process began causing back-pressure up the replication line of Teem databases. This in turn caused normal and overflow disk space to be filled and eventually led to a primary database becoming unavailable and the service disruption that followed. The failing process was identified and corrected, the blockage released and the systems were brought back online at 1020 MT. Data and replication backlogs will continue to work their way down throughout the weekend and progress will continue to be monitored.
We apologize for any inconvenience this downtime may have caused and thank you for your continued use of the Teem platform.
Posted Jan 25, 2020 - 12:05 MST
Monitoring
Service has been restored and Teem is monitoring results.
Posted Jan 25, 2020 - 10:25 MST
Identified
Teem has identified the issue and is working to restore service.
Posted Jan 25, 2020 - 09:07 MST
This incident affected: Web Interface, Mobile Data, API, EventBoard, and LobbyConnect.