Teem Detailed Root Cause Analysis – Severity 2 – June 5, 2023
Outlook and Google Events are not Syncing from Teem to Source Calendar
Description:
On June 5, 2023, at approximately 11:43am MST, customers support received additional reports of “Outlook and Google Calendar Events are not Syncing from Teem to Source Calendar”. An event created in Teem should sync back to the source calendar and vice versa.
Type of Event:
Performance Degradation
Services\Modules Impacted:
Production (Reservation Module)
Timeline:
On June 5, 2023, at approximately 11:43am MST, customer support received additional reports of “Outlook and Google Calendar events are not Syncing from Teem to Source Calendar”. The initial ticket was escalated to a S2 incident at 11:43am MST and posted to our status page alerting customers. Internal teams continue to investigate this issue through June 6, 2023. Our internal team identified the root cause of the issue at 9:10am MST and began work on the resolution. The status page was updated from investing to an identified phase. At 3:30pm MST a hotfix was implemented, and the status page was moved to a monitoring phase. At 4:27pm MST after the hotfix was implemented, internal teams acknowledged that calendar events are slowly being processed through the queue. The internal teams continue to monitor the queue through June 7, 2023. At 6:56am MST, all prior events created before the hotfix was implemented have been cleared. Internal teams continue to monitor new events that were created after the hotfix was released. As monitoring continued, customers reported that all calendar events are syncing at about 4:02pm MST. At 5:15pm MST, the incident was marked as resolved.
Total Duration of Event:
2 days 5 hours 32 mins
Root Cause Analysis:
Engineering teams found this was related to a release pushed out May 19, 2023.
Preventative Action:
While the engineering team fixed the regression bug. They found that there was an internal process during the release roll out that needed improvement with QA.