Our workarounds have been deployed successfully and performance has returned to normal. Both User Search and User Count endpoints have been re-enabled and all Pronto functionality is restored. At this point we believe the root cause was a database-level bug. We are working with our database vendor to confirm the bug and get a permanent fix. In the meantime, now that we know how to workaround it, we expect no further problems from this.
Posted Dec 28, 2021 - 20:09 MST
Update
We are continuing to work on a fix for this issue. We have deployed some workarounds and seen some promising results. We are working to identify and address the remaining slow areas using a similar approach. We will provide another update after testing and deploying those additional workarounds.
Posted Dec 28, 2021 - 18:26 MST
Identified
We've identified the very slow queries and are attempting some workarounds to speed them up. It is still unclear what the root cause of the slow queries is, but we are hopeful that this workaround will resolve the immediate issues to get performance back to normal and allow us to reenable all endpoints.
Posted Dec 28, 2021 - 15:59 MST
Update
We have temporarily disabled two endpoints that are causing the issue in order to protect the rest of the application. These two endpoints are:
1. User search - Attempting to search for a user in any context will not work. This includes starting a new DM (existing DMs are unaffected), adding new users to a group, or searching for users in org management. 2. User counts - In org management the overall user count will not be displayed
We apologize for this loss of functionality, but deemed it necessary in order to prevent further issues across the Pronto app. We are working with our database vendor directly to diagnose and solve this issue as quickly as possible. Thanks for your patience.
Posted Dec 28, 2021 - 13:53 MST
Investigating
We are currently seeing degraded performance on user-related queries in the main Pronto database. User searches in org management and also in the client apps are currently taking a long time or timing out. User online status is also affected and may not be reflecting the correct state right now. We are actively investigating and will update as soon as we know more.
Posted Dec 28, 2021 - 13:31 MST
This incident affected: Pronto Platform and Pronto Web App.