Early this morning we were alerted to issues with calendar syncing. Looking through our logs we we able to quickly confirm that there was an issue and that it was restricted to o365.

The logs revealed that we were being rate-limited by o365.

This was caused by an unusual influx of updated pushes from a subset of o365 customers. This situation is normally handled gracefully, but due to certain characteristics of the pushes we began making too many requests to o365. This trickeled out to impact all o365 customers.

Once we identified the source of the issue. We paused and reset the pushes for o365, this is required to give time for the rate-limiting to be reset. Once it was reset, we re-enabled the pushes for o365 and monitored it to ensure that the influx of pushes was not going to reoccur. Everything is now stable and working as excepted.

Note, this issue did not impact calendar sync for any on-prem Exchange servers nor for Google calendars.