Teem Detailed Root Cause Analysis | 4.05.2024

404 Error Site Wide Outage

‌

We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.

‌

Description:

On April 5th, 2024,  internal teams and customer support noticed issues happening with logging and being able to access sites. Reports of some Teem instances experiencing inability to login with a forward of a 404 error also were presented. This issue impacted a large scale of Teem Customers.

This cause of this site outage was due to a migration to a new server that would allow easier rollbacks and data integrity, that unfortunately had a performance error and needed to be done early.

Upon receiving notice of the site access issues, our dedicated team promptly took action to restart the migration process, ensuring its successful completion and resolving the issue.

Type of Event:

Site Access

Remediation:

Upon immediate notice of the login issues, our dedicated Teem team was able to restart the migration process to successful completion, resolving the issue.

Root Cause Analysis:

There was a production outage caused by a database that we were hosting on. We needed to then move the data base to our standby server and set up a new one. We had planned the migration for a few weeks later and were preparing for it but the server encountered errors before the planned maintenance happened forcing us to start the migration early. We started the migration server backed up the DB and re-established connection with services.

Preventative Action:

The Teem team was able to start a migration process and monitor it through successful completion and review with internal testing to ensure the issue was resolved for customers. This incident has been closed, but our team is dedicated to closely monitoring future updates as they are released to ensure the best customer experience.