Knowledgebase Article Number |
360050986553 |
Description |
Published June 29th, 2020, at approximately 3:00 p.m. EST. |
Resolution |
At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on March 14th and 16th, 2020 described below. Summary: Friday, June 26th 08:02 AM EST 08:20 AM EST 08:24 AM EST 08:26 AM EST 08:35 AM EST T1 then stopped and started ExactaExportAdapter and ExactaExport services, but still received the same error as above. 08:50 AM EST 09:05 AM EST 09:05 AM EST 09:11 AM EST 09:22 AM EST 09:27 AM EST 09:50 AM EST 10:05 AM EST 10:14 AM EST 10:43 AM EST 11:04 AM EST 11:23 AM EST
Root Cause: After Bastian's DBA, along with the support team, determined that the issues were being caused by a full tempdb database, several members of Bastian's software division investigated into why the database filled up. The "Update Daily Stats for Wave" job was originally created to pre-compute Inmar's dashboard which was intended to lighten the load on the database, so that individual Portal sessions did not pull an intense query. This query (which previously was being ran each time that a user viewed their dashboard) normally runs every five minutes, and takes between 30-90s to complete, but the job that started at 1:30p on 6/25 got hung unexpectedly, and never ended. This caused the tempdb database to slowly fill up as the query kept taking more and more resources, and ultimately is what caused the database to reach capacity. Next Steps and Preventive Actions: Bastian's software team was able to optimize the existing query to greatly improve performance and efficiency, and also updated the schedule of when the query will be run, from every five minutes to ten minutes. This will lighten the overall load on the database, and will provide the query plenty of time to complete, which should prevent this from happening in the future.
Our commitment to you: Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.
|
Comments
0 comments