Reported O365 Calendar Sync Issue
Incident Report for Teem
Postmortem

Our investigation found two distinct issues:

The first has to do with push notifications received from Office 365 that contain instructions that Teem should renew our subscription. These unique push notifications would trigger Teem to start a new subscription to changes on the calendar folder. However, it was found that Office 365 would repeatedly send these push notifications and that in certain cases this could create a race condition where the subscription saved to Office 365 was different than that saved in Teem. To remediate this, we have implemented a scheduling system for calendar jobs that lets Teem react after a short time to the first of these unique push notifications, and ignore any subsequent notifications. Monitoring has shown that this change is effective in renewing our calendar subscription and removing these race conditions.

The second issue had to do with normal errors that occur when working with Office 365. When Teem received these errors, after trying multiple times over the course of a few minutes, we would stop attempting to sync the calendar. While most calendars naturally recovered because of a subsequent calendar change, a subset of calendars would not have a later change, and thus stay out of sync. Our investigation showed that some of these errors were temporary. Using the scheduling system described above, we now isolate temporary errors and schedule a retry for a later time. This helps ensure that we can sync in the events as expected.

Posted Oct 09, 2019 - 09:46 MDT

Resolved
This incident has been resolved. We have continued monitoring the calendar-related update, and systems are working as expected.
Post mortem to be posted by Thursday, October 10th.
Posted Oct 03, 2019 - 09:40 MDT
Update
We have continued monitoring the calendar-related update released September 19th, and O365 calendars continue working as expected, except due to the non-calendar specific database event reported last Friday.

We will continue monitoring this evening and will provide another update by 10 AM MT/12 PM ET tomorrow.
Posted Oct 02, 2019 - 13:45 MDT
Update
We are continuing to monitor the process and will release update for any additional issues.
Posted Sep 28, 2019 - 16:11 MDT
Update
Teem is continuing to monitor the process and released update for any additional issues.
Posted Sep 25, 2019 - 13:47 MDT
Monitoring
An update has been deployed to production. We are monitoring to see if this update resolves the calendar sync issues.
Posted Sep 20, 2019 - 09:47 MDT
Identified
Engineering has identified the root cause, and are implementing a plan for resolution. We will provide another update as soon as the changes have been made.
Posted Sep 17, 2019 - 09:35 MDT
Update
Engineering believes they have identified the root cause and they are currently verifying. We will provide another update as soon as the root cause is confirmed.
Posted Sep 16, 2019 - 11:13 MDT
Update
Teem is continuing to investigate the root cause of this issue. Additional logging has been implemented to assist with this effort.
Posted Sep 13, 2019 - 15:03 MDT
Update
Teem is continuing to investigate the root cause of the issue. In the interim:
1. Any O365 calendar found to be out of sync will automatically be re-synced every 8 hours
2. If you encounter a calendar which is out of sync, you can follow these instructions to manually initiate a re-sync: https://www.teem.com/support/eventboard-room-scheduling/re-syncing-calendars?from_search=36544921
Posted Sep 11, 2019 - 18:25 MDT
Investigating
Teem is investigating a reported calendar sync issue affecting multiple O365 customers. Affected calendars may fail to sync with O365 until a manual re-sync is performed.
Posted Sep 11, 2019 - 12:02 MDT
This incident affected: Exchange Sync.