Emergency Database Maintenance
Scheduled Maintenance Report for Teem
Postmortem

Description
As a fast follow on to the work completed for the critical database outage the previous evening (9/24). Additional configuration of the database servers was required to ensure more reliable data replication moving forward.

Root Cause/Remediation

The initial configuration of the new standby database system and replication method were suboptimal and through monitoring it was identified that load on the system started to cause replication delay and needing immediate resolution. The problems were causing excessive local WAL transfer and storage and the potential for less reliable replication. The standby was brought up-to-date and the database configuration was updated to ensure more reliable replication and availability.

Posted 8 months ago. Sep 28, 2018 - 09:09 MDT

Completed
As a fast follow on to the work completed for the critical database outage in the evening of 9/24 (MST). Additional configuration of the replication servers and services were required to ensure high availability of the data moving forward.
Posted 8 months ago. Sep 25, 2018 - 23:13 MDT
Verifying
Verification is currently underway for the maintenance items.
Posted 8 months ago. Sep 25, 2018 - 22:44 MDT
Update
Teem is continuing the maintenance work and will provide additional updates when available.
Posted 8 months ago. Sep 25, 2018 - 22:04 MDT
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted 8 months ago. Sep 25, 2018 - 21:00 MDT
Scheduled
We need to shedule emergency database maintenance that will cause disruption to services at 9PM tonight (Tuesday, September 25th). We apologize for this inconvenience.
Posted 8 months ago. Sep 25, 2018 - 20:35 MDT
This scheduled maintenance affected: Web Interface, Mobile Data, API, Google Apps Calendar, and Exchange Sync.