Since our last update we have been carefully monitoring notification delays and the numbers look great and have stayed that way for about 4 hours now. We believe that this issue is now fully resolved and will close this incident. The root cause was that a database upgrade appears to have reset some of the optimizations that were used to ensure fast database access. After reconfiguring the database with the proper optimizations, performance returned to normal levels.
Posted Jul 22, 2022 - 18:30 MDT
Monitoring
After making some more database optimizations all our numbers are looking good again. We will continue monitoring for the next few hours, but for now notifications are back to normal delivery times.
Posted Jul 22, 2022 - 13:57 MDT
Investigating
We are once again experiencing delayed notifications, due to some slow database queries. We are working again with our vendor to investigate and will update here as we learn more. We sincerely apologize for the ongoing issue. The team is working diligently to solve this issue for good.