Teem Detailed Root Cause Analysis – Severity 2 – 08/29/2023

Description:

On August 29, 2023, at approximately 4:45am EDT, customer support started to receive reports of Calendar Events not syncing at all. When customers are booking events, Teem and Source Calendars are not syncing causing disruption in customer workflow.

Type of Event:

Service Disruption

Services\Modules Impacted:

Production – Calendars

Timeline:

On August 29, 2023, at approximately 4:45am EDT, customer support began to receive reports of Calendar Events not syncing at all. When customers are booking events, Teem and Source Calendars are not syncing causing disruption in customer workflow. At approximately 10:26am EDT, the engineering team identifies the issue and begins to implement the fix. At 10:36am EDT all customers are made aware that the fix has deployed to production and that we are moving into a monitoring phase. Customers begin to confirm the fix and at 11:01 EDT, the status page is marked as resolved.

‌

Total Duration of Event:

6hrs 16mins

‌

Root Cause:

The team continues to narrow down the root cause. However, the DevOps team identifies an internal configuration issue and begins to apply a manual fix.

Preventative Action:

Our internal teams are working on enhancing the observability to bring awareness to the appropriate teams.