Description

On October 11 Teem received and escalated reports that a subset of devices were not receiving calendar event updates. Booking an event on the calendars associated to the device was successful through Outlook, Google or other calendar clients, however the events were not being received by Teem nor updated to the devices. As Teem investigated it became clear that this was primarily affecting several Office365 or Exchange hosted calendars - specifically calendars with a large number of scheduled events and/or recurring events. Two separate hot-fixes were deployed on October 18 to address the issues found with Microsoft’s Exchange Web Service (EWS) API. After a week of monitoring the fixes this incident was closed.

‌

Root Cause / Remediation

Teem identified two issues/limitations with EWS APIs provided by Microsoft for Exchange and O365 when working with large calendars (i.e. calendars with many individual or recurring events):

• Calendars with a large number of events when queried by the EWS API would timeout on O365 hosted servers or the client’s on-premise Exchange server.
• API queries to determine the originating meeting in a recurring meeting set become extremely non-performant in the EWS API with a large number of recurring meetings/recurring meeting instances

Teem has changed our approach to account for these EWS API limitations and is now utilizing concurrent batching of requests to gather event information for calendars and adding Teem-side caching to minimize usage of expensive API queries via EWS. These changes have facilitated proper event synchronization and improved performance of these operations substantially.