This incident has been resolved. All systems are healthy.
If you have devices that are still on 2.10.5, please update to 2.10.6 as soon as possible. If you need any assistance, please open a Support ticket and we'll be happy to assist.
Posted Oct 15, 2019 - 13:38 MDT
Update
We are continuing to see improvements across all systems due to the movement off of iOS version 2.10.5 to 2.10.6. Response times from Teem's servers are now within a normal range.
All devices (iOS, Android, and Crestron) using Eventboard are expected to be able to authenticate and/or activate. There may be periods of intermittent slowness, if that persists please open a Support ticket for further troubleshooting.
We will continue to monitor and will update again tomorrow by 3 PM MT/5 PM ET.
Posted Oct 14, 2019 - 16:25 MDT
Update
With the release of iOS 2.10.6 we have seen the number of devices on 2.10.5 drop significantly, and it continues to do so. Due to the movement off of 2.10.5 to 2.10.6, we have seen a large drop in inbound reports of devices unable to authenticate and/or activate across all devices (iOS, Android, and Crestron), and we are currently following up on previous reports.
Teem systems are trending back towards normal. There may still be intermittent slowness but it is anticipated that will continue to improve over the next few hours.
Next update will be by 5 PM MT/7 PM ET
Posted Oct 14, 2019 - 13:10 MDT
Update
Eventboard iOS version 2.10.6 for Eventboard has been approved and has begun propagating, which can take several hours before it appears in the app store. If the iOS device has 'auto-update' enabled, the Eventboard app will update to 2.10.6 when it is available.
Important to note: iOS Devices on Eventboard 2.10.5 should be updated to 2.10.6 as soon as it is available. For iOS devices on 2.10.4 or lower, updating to 2.10.6 is recommended for the additional features and fixes, but it isn't essential to fixing the connectivity/syncing issues.
Additionally, Teem has implemented changes on the back end to block traffic from devices on iOS version 2.10.5 and to free up resources and improve performance on all other iOS versions and platforms (Android and Crestron).
Next Update will be by 2PM MT/4PM ET
Posted Oct 14, 2019 - 10:18 MDT
Update
For all devices that are not on iOS 2.10.5, which includes all Crestron and Android devices, we are seeing many more devices that are now able to activate and authenticate. However, some devices are still appearing offline or having intermittent issues.
For devices on iOS 2.10.5, they continue showing as offline. When 2.10.6 is released in the App Store and these devices update, they will begin coming back online.
Next Update by 12PM MT/2PM ET
Posted Oct 14, 2019 - 07:59 MDT
Monitoring
We have deployed back-end changes and are seeing the load on Teem’s servers decrease significantly. The expected behavior for all devices that are not on iOS 2.10.5, which includes all Crestron and Android devices, is that they can now activate and authenticate as normal. However, some intermittent issues may also be seen.
For devices running iOS 2.10.5 the expected behavior is that there will be slowness in activating, and also some devices may continue reporting as offline. Until 2.10.6 is available in the App Store this behavior is expected to continue.
Next Update: 8AM MT/10AM ET
Posted Oct 14, 2019 - 01:59 MDT
Update
iOS version 2.10.6 is not yet available in the app store. As such, we are continuing to work on a workaround that will rate limit traffic from iOS devices on 2.10.5.
Next Update by 2AM MT/4 AM ET
Posted Oct 13, 2019 - 23:56 MDT
Update
Both Eventboard and LobbyConnect are continuing to have issues authenticating with Teem servers on all devices (Android, iOS, Crestron).
Eventboard version 2.10.6 has yet to be released in the iOS app store. We are going to continue looking into back-end changes to our environment that will help until 2.10.6 is released.
To re-state the issue, it is caused by Eventboard iOS version 2.10.5. However, all versions and devices are affected as it is causing increased load on Teem's servers. When 2.10.6 is released and devices on 2.10.5 begin upgrading to 2.10.6 the load on Teem's servers will decrease, and this will lead to normal load times.
Next Update in 4 hours at Midnight MT/2 AM ET
Posted Oct 13, 2019 - 19:57 MDT
Update
Both Eventboard and LobbyConnect are continuing to have issues authenticating with Teem servers. The fix, which is included in version 2.10.6, is still pending release in the app store. We anticipate it will be available sometime today. When it is released, if your iOS device is set to auto-update apps then Eventboard will be updated automatically as soon as 2.10.6 is available in the app store. Next Update: Today by 8 PM MT/10 PM ET
Posted Oct 13, 2019 - 08:47 MDT
Update
Both Eventboard and LobbyConnect are continuing to have issues authenticating with Teem servers, this is due to the queuing up of requests. To be clear, this behavior will continue until Eventboard 2.10.6 on iOS is released in the app store and devices running 2.10.5 are updated with the new code.
We are still awaiting confirmation that version 2.10.6 on iOS has been made available in the app store. Next Update: Tomorrow, Sunday, October 13th by 10 AM MT/12 PM ET
Posted Oct 13, 2019 - 00:39 MDT
Update
Eventboard 2.10.6 on iOS has been submitted to Apple and is pending approval. Next Update: Tomorrow, Sunday, October 13th by 10 AM MT/12 PM ET
Posted Oct 12, 2019 - 22:37 MDT
Identified
We have identified the issue. Eventboard iOS version 2.10.5, the latest release, included a race condition which increased calls made to our servers. This additional load led to increased latency and device connectivity issues. We have developed a fix, and are in process of implementing this in iOS version 2.10.6. Only iOS devices on 2.10.5 will need to be updated. Note: Crestron and Android devices will not need to be updated Next Update: Tonight by 11:30 PM MT/1:30AM ET
Posted Oct 12, 2019 - 19:29 MDT
Update
As part of the investigation, it was found app.teem.com was timing out, this has been addressed and app.teem.com is loading as normal. We are continuing to investigate the Eventboard & LobbyConnect device syncing issues. The next update will be by 8 PM MT/10 PM ET this evening.
Posted Oct 12, 2019 - 18:00 MDT
Update
We have verified that calendar events are unaffected, and are syncing successfully. The current issue appears to be specific to devices. The next update will be by 8 PM MT/10 PM ET this evening.
Posted Oct 12, 2019 - 16:55 MDT
Investigating
We are seeing connectivity issues with Eventboard and Lobbyconnect. We are currently investigating. Next update will be by 8 PM MT/10 PM ET this evening.