Knowledgebase Article Number
Published January 27, 2021, at approximately 2 p.m. EST.
At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on Jan 21st – Jan 22nd, 2021 described below.
5:55 PM EST
Jose from Inmar called in at 5:55 pm stating that their Sorter Stats report was down and also added that Exacta was down. Foster Akado looked into the issue.
6:43 PM EST
Foster reached out to escalation who then reached out to Larry Marshall (SQL Dev)
7:34 PM EST
Larry corrected the Archive & Delete issues and Jose from Inmar confirmed that everything was working. The issue with Delete and Archive not completing had caused the Tempdb database to fill up.
8:04 PM EST
Jose called in to report another issue, which was the Wave Management>View All Pick Orders>Open Orders utility continuously spinning and then erroring out.
8:26 PM EST
8:26 PM EST
Krystal Nolan was pulled in to assist Foster in communicating with Inmar and providing them with updates.
10:16 PM EST
10:28 PM EST
10:54 PM EST
2:00 AM EST
Larry has made numerous changes to help fine-tune the database. However, there continue to issue processing the wave.
3:00 AM EST
4:00 AM EST
The cause of this issue was a combination of a specific wave having a large number of orders and a database query being inefficient when running against that wave.
Larry provided script updates so that Bastian could temporarily optimize the Wave Management page for the problem wave. This would allow them to perform every action need to close out the wave. After this was done and that wave was processed, Bastian would revert the script back to normal so that they could process everything else.
Long term fix was the Database team has provided an update to one of our maintenance scripts to make it more efficient. This should prevent delete and archive from hanging up. Also, the Database team updated the view used by Wave Management so that it is optimized and runs faster.
Next Steps and Preventive Actions:
Continue monitoring database performance.
Our commitment to you:
Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.