On 6 January, as usage of Teem services increased for the day, there was an increase of duplication errors on our database. This increase caused a delay in replication, which in turn caused a greater number of duplication errors. This negative feed forward mechanism caused a degraded site performance. Reduced load on the database by scaling down our service instances allowed the replication to catch up, and remain health. Going forward, there will be database maintenance on the specific tables that exhibit these issues to guard against this in the future.
Posted Jan 28, 2020 - 14:25 MST
Resolved
Performance continues to be normal. This incident is now resolved.
Posted Jan 06, 2020 - 16:01 MST
Monitoring
A fix has been implemented, and all services are working as normal. We will monitor through the rest of the day. Next update will be by 5:00pm MT/7:00pm ET
Posted Jan 06, 2020 - 13:52 MST
Investigating
We’re currently seeing loading issues with app.teem.com and related services such as activating devices and signing into the mobile app. Our team is working to identify the root cause and implement a solution. Next update will be by 3:00pm MT/6:00pm ET
Posted Jan 06, 2020 - 12:45 MST
This incident affected: Web Interface and Mobile Data.