o365 calendar syncing outage
Incident Report for Teem

Early this morning we were alerted to issues with calendar syncing. Looking through our logs we we able to quickly confirm that there was an issue and that it was restricted to o365.

The logs revealed that we were being rate-limited by o365.

This was caused by an unusual influx of updated pushes from a subset of o365 customers. This situation is normally handled gracefully, but due to certain characteristics of the pushes we began making too many requests to o365. This trickeled out to impact all o365 customers.

Once we identified the source of the issue. We paused and reset the pushes for o365, this is required to give time for the rate-limiting to be reset. Once it was reset, we re-enabled the pushes for o365 and monitored it to ensure that the influx of pushes was not going to reoccur. Everything is now stable and working as excepted.

Note, this issue did not impact calendar sync for any on-prem Exchange servers nor for Google calendars.

Posted 4 months ago. Jun 28, 2017 - 10:51 MDT

Resolved
o365 calendar syncing is stable and working as expected.
Posted 4 months ago. Jun 28, 2017 - 10:46 MDT
Monitoring
We have identified an issue that is currently restricted to o365 calendar syncing. The issue was caused by an overactive sync pushes from o365. We have resolved the push issue and are closely monitoring calendar syncing. At this point calendar syncing should occur as normal for all calendar systems.
Posted 4 months ago. Jun 28, 2017 - 09:00 MDT