Degraded performance: app.teem.com and other services

Incident Report for Teem

Postmortem

On December 16th the Teem platform received a large spike in traffic from Google Calendar provider which ultimately created a large backlog of asynchronous platform jobs. The systems over balanced these jobs causing heavy load on the database. Tuning the asynchronous capacity for the related workloads helped clear the backlog and prevent further issues. Additional database maintenance was performed in late January to further reduce the possibility of recurrence.

Posted Feb 07, 2020 - 16:56 MST

Resolved

After monitoring throughout the weekend this incident has been resolved

Posted Dec 16, 2019 - 08:55 MST

Monitoring

Performance has been normal for app.teem.com, EventBoard, LobbyConnect and all other apps/services since the last update.

Teem has released an update to help prevent degraded performance in this scenario in the future and will continue to monitor throughout the day, at which time we will resolve the incident.

Posted Dec 13, 2019 - 11:22 MST

Update

Performance has been normal for app.teem.com, EventBoard, LobbyConnect and all other apps/services since the last update.

Teem is preparing an update to handling and processing systems (currently in testing) and will deploy when complete.

Next update will be as status changes or tomorrow by 9am MT/11am ET

Posted Dec 12, 2019 - 12:38 MST

Identified

Performance has been normal for app.teem.com, Eventboard, LobbyConnect and all other apps/services since the last update.

Teem has identified an abnormal spike in change notifications for certain calendar providers and is in the process of updating handling and processing systems to account for the change in behavior.

Next update will be as status changes or tomorrow by 9am MT/11am ET

Posted Dec 11, 2019 - 09:25 MST

Update

Performance has returned to normal for app.teem.com, Eventboard, LobbyConnect and all other apps/services are also fully functional.

The investigation actively continues and is focused on finding and addressing the reason for the increased load yesterday and today.
Next update will be tomorrow by 9am MT/11am ET

Posted Dec 10, 2019 - 17:41 MST

Update

We are continuing to investigate the increased load on the system and its cause.
Performance continues to be impacted with extended periods of slowness or failing to load applications.
Next update will be by 6pm MT/8pm ET

Posted Dec 10, 2019 - 14:55 MST

Update

We are continuing to investigate the cause of the increased load on our systems. Performance continues to be impacted with extended periods of slowness or failing to load applications.
Next update: 3pm MT/5pm ET

Posted Dec 10, 2019 - 12:21 MST

Investigating

We’re currently experiencing a recurrence of the service disruption, with reports of loading slowness with app.teem.com, EventBoard, LobbyConnect, as well as other services such as device registrations.
Our team is actively investigating.
Next update will be by 12:30pm MT/2:30pm ET

Posted Dec 10, 2019 - 10:19 MST

Update

Performance remains normal. We have made adjustments to our back-end systems and are continuing to monitor the results through the day.
Next update today by 4pm MT/6pm ET

Posted Dec 10, 2019 - 08:52 MST

Monitoring

Performance has returned to normal for app.teem.com, Eventboard, LobbyConnect and all other apps/services are also fully functional.
We have not identified the root cause yet, but are continuing to investigate and monitor.
Next update will be tomorrow by 9am MT/11am ET

Posted Dec 09, 2019 - 18:00 MST

Update

We are continuing to investigate the cause of the issue. The system is catching back up and performance is gradually improving
Next update will be by 7pm MT/9pm ET

Posted Dec 09, 2019 - 16:18 MST

Investigating

We’re currently experiencing a service disruption with reports of loading issues with app.teem.com, EventBoard, and LobbyConnect.
Our team is working to identify the root cause and implement a solution.
Next update will be by 4:30pm MT/6:30pm ET

Posted Dec 09, 2019 - 14:39 MST

This incident affected: Web Interface and EventBoard.

Subscribe to Incident