Office 365 Calendar Sync
Incident Report for Teem
Postmortem

Description

On October 18 this issue was separated from a previous incident affecting calendar synchronization for a subset of Office365 client's calendars and escalated for immediate research/remediation. Client reports were of devices sporadically not receiving new events, however forcing a synchronization or renewing a subscription/watch would temporarily resolve the issue. After additional testing and research an adjustment to the frequency and timing of Teem’s automated mechanism to validate and renew these subscriptions/watches was made on October 22. All subscriptions were updated and affected clients were contacted on October 23 after 24 hours of successful monitoring.

Root Cause / Remediation

Office365 via the EWS API will for a variety of reasons, periodically expire subscriptions and Teem has created mechanisms to automatically validate and renew these. Receiving notifications via these subscriptions is how Teem triggers synchronization operations so that new/changed/removed event information is delivered to all Teem tools and devices. Teem adjusted the frequency and algorithms used to ensure valid watches to more aggressively force renewals as necessary and affected calendars have been operating successfully since that time.

Posted Oct 30, 2018 - 12:45 MDT

Resolved
The issue has been resolved. Teem has updated the frequency which we force subscription/watch renewal for O365 calendars.
Posted Oct 24, 2018 - 11:39 MDT
Update
We have identified a suspected root cause for this problem and our engineering team is more closely monitoring the calendar sync status of Office 365 customers to collect additional, detailed data to assist in resolving this problem. We hope to have a permanent resolution in place as soon as we are able.
Posted Oct 22, 2018 - 16:25 MDT
Identified
We have identified the core issue that is causing our system not to get updates from Office365. Our engineers are investigating a resolution for this issue and hope to have a fix as soon as possible.
Posted Oct 19, 2018 - 13:04 MDT
Investigating
A subset of our customers using Office 365 is seeing intermittent issues where devices are not being updated with room calendar events, subsequently eventboard devices appear to have inaccurate information. For these customers, the Teem Dashboard calendars “Last Sync” date appears to be incorrect. Our engineering team is dedicated to identifying and resolving this problem as quickly as possible and are working closely with our Customer Support team to expedite resolution.
Posted Oct 18, 2018 - 15:25 MDT
This incident affected: Exchange Sync.