Teem service performance and calendar synchronization issues

Incident Report for Teem

Postmortem

Production Performance Issues - Postmortem

Root Cause:

database maintenance caused queries to be under-optimized
increased total available connections caused additional load on the database

Problem Resolution:

reduced total available connections on the front end
updated database software and apply settings changes
ran maintenance to improve query performance

Future Improvements:

further scale database to handle more requests
rely more on caching of requests

Posted Aug 11, 2017 - 15:32 MDT

Resolved

All systems should be operational.

Posted Aug 11, 2017 - 15:06 MDT

Update

Database maintenance seems to have helped overall performance. We will continue to work on further improvements for site performance.

Posted Aug 11, 2017 - 09:18 MDT

Update

We are planning on running database maintenance tonight that should improve the performance of the system. This maintenance is scheduled to begin at 12PM MDT.

Posted Aug 10, 2017 - 19:14 MDT

Update

We just released a hotfix that should help with performance issues. We are still monitoring the performance of the site and the performance issues have improved - but are not fully resolved.

Posted Aug 10, 2017 - 16:10 MDT

Identified

Teem is currently investigating an issue that may be affecting some customers in the following areas:
- Calendar sync may be experiencing delayed updates
- The Teem dashboard is experiencing slower load times than usual
- Online devices may be alerting as offline and the admin dashboard may not accurately reflect health status

Teem is actively working on this issue and will continue to update this page as we have more information.

Posted Aug 10, 2017 - 11:54 MDT