Teem service performance and calendar synchronization issues
Incident Report for Teem
Postmortem

Production Performance Issues - Postmortem

Root Cause:

  • database maintenance caused queries to be under-optimized

  • increased total available connections caused additional load on the database

Problem Resolution:

  • reduced total available connections on the front end

  • updated database software and apply settings changes

  • ran maintenance to improve query performance

Future Improvements:

  • further scale database to handle more requests

  • rely more on caching of requests

Posted Aug 11, 2017 - 15:32 MDT

Resolved
All systems should be operational.
Posted Aug 11, 2017 - 15:06 MDT
Update
Database maintenance seems to have helped overall performance. We will continue to work on further improvements for site performance.
Posted Aug 11, 2017 - 09:18 MDT
Update
We are planning on running database maintenance tonight that should improve the performance of the system. This maintenance is scheduled to begin at 12PM MDT.
Posted Aug 10, 2017 - 19:14 MDT
Update
We just released a hotfix that should help with performance issues. We are still monitoring the performance of the site and the performance issues have improved - but are not fully resolved.
Posted Aug 10, 2017 - 16:10 MDT
Identified
Teem is currently investigating an issue that may be affecting some customers in the following areas:
- Calendar sync may be experiencing delayed updates
- The Teem dashboard is experiencing slower load times than usual
- Online devices may be alerting as offline and the admin dashboard may not accurately reflect health status

Teem is actively working on this issue and will continue to update this page as we have more information.
Posted Aug 10, 2017 - 11:54 MDT