Teem service performance and calendar synchronization issues
Incident Report for Teem

Production Performance Issues - Postmortem

Root Cause:

  • database maintenance caused queries to be under-optimized

  • increased total available connections caused additional load on the database

Problem Resolution:

  • reduced total available connections on the front end

  • updated database software and apply settings changes

  • ran maintenance to improve query performance

Future Improvements:

  • further scale database to handle more requests

  • rely more on caching of requests

Posted 2 months ago. Aug 11, 2017 - 15:32 MDT

Resolved
All systems should be operational.
Posted 2 months ago. Aug 11, 2017 - 15:06 MDT
Update
Database maintenance seems to have helped overall performance. We will continue to work on further improvements for site performance.
Posted 2 months ago. Aug 11, 2017 - 09:18 MDT
Update
We are planning on running database maintenance tonight that should improve the performance of the system. This maintenance is scheduled to begin at 12PM MDT.
Posted 2 months ago. Aug 10, 2017 - 19:14 MDT
Update
We just released a hotfix that should help with performance issues. We are still monitoring the performance of the site and the performance issues have improved - but are not fully resolved.
Posted 2 months ago. Aug 10, 2017 - 16:10 MDT
Identified
Teem is currently investigating an issue that may be affecting some customers in the following areas:
- Calendar sync may be experiencing delayed updates
- The Teem dashboard is experiencing slower load times than usual
- Online devices may be alerting as offline and the admin dashboard may not accurately reflect health status

Teem is actively working on this issue and will continue to update this page as we have more information.
Posted 2 months ago. Aug 10, 2017 - 11:54 MDT