Subset of devices unable to receive calendar event updates
Incident Report for Teem
Postmortem

Description

On October 11 Teem received and escalated reports that a subset of devices were not receiving calendar event updates. Booking an event on the calendars associated to the device was successful through Outlook, Google or other calendar clients, however the events were not being received by Teem nor updated to the devices. As Teem investigated it became clear that this was primarily affecting several Office365 or Exchange hosted calendars - specifically calendars with a large number of scheduled events and/or recurring events. Two separate hot-fixes were deployed on October 18 to address the issues found with Microsoft’s Exchange Web Service (EWS) API. After a week of monitoring the fixes this incident was closed.

Root Cause / Remediation

Teem identified two issues/limitations with EWS APIs provided by Microsoft for Exchange and O365 when working with large calendars (i.e. calendars with many individual or recurring events):

  • Calendars with a large number of events when queried by the EWS API would timeout on O365 hosted servers or the client’s on-premise Exchange server.
  • API queries to determine the originating meeting in a recurring meeting set become extremely non-performant in the EWS API with a large number of recurring meetings/recurring meeting instances

Teem has changed our approach to account for these EWS API limitations and is now utilizing concurrent batching of requests to gather event information for calendars and adding Teem-side caching to minimize usage of expensive API queries via EWS. These changes have facilitated proper event synchronization and improved performance of these operations substantially.

Posted 7 months ago. Oct 30, 2018 - 12:23 MDT

Resolved
The two updates released as hot-fixes last week have resolved all the known instances of devices failing to update with any calendar events from O365 and Exchange.
Posted 7 months ago. Oct 24, 2018 - 11:32 MDT
Monitoring
We have released several hotfixes to production that should address some of the issues related to calendar sync issues. These fixes have been verified in several instances to resolve issues for customers using Exchange servers. Clients using Office365 are still experiencing sporadic calendar sync issues and we are creating an incident specifically for these issues. We will continue to monitor for other calendar sync issues related to Exchange to see if those have been definitely resolved.
Posted 7 months ago. Oct 18, 2018 - 15:14 MDT
Identified
Teem has identified an issue/limitation with APIs provided by Microsoft for Exchange and O365 when working with very large calendars. Our teams are working on a potential resolution that will be deployed as a hotfix once validated internally.
Posted 7 months ago. Oct 12, 2018 - 16:46 MDT
Investigating
Teem has received reports that a subset of devices have not received calendar event updates. Booking an event on these calendars is successful through Outlook, Google or other calendar clients, however the events are not received by Teem nor updated to the devices. The engineering team is investigating potential causes.
Posted 8 months ago. Oct 11, 2018 - 20:28 MDT
This incident affected: Exchange Sync.