Knowledgebase Article Number

1500002033541

Description

Published January 27, 2021, at approximately 2 p.m. EST. 
Last updated on January 27, 2021.

Resolution 

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on Jan 21st – Jan 22nd, 2021 described below.
 

Summary:

5:55 PM EST

Jose from Inmar called in at 5:55 pm stating that their Sorter Stats report was down and also added that Exacta was down. Foster Akado looked into the issue.

6:43 PM EST

Foster reached out to escalation who then reached out to Larry Marshall (SQL Dev)

7:34 PM EST

Larry corrected the Archive & Delete issues and Jose from Inmar confirmed that everything was working. The issue with Delete and Archive not completing had caused the Tempdb database to fill up.

8:04 PM EST

Jose called in to report another issue, which was the Wave Management>View All Pick Orders>Open Orders utility continuously spinning and then erroring out.

8:26 PM EST
Foster Akado escalated to Evan Sturgis

8:26 PM EST

Krystal Nolan was pulled in to assist Foster in communicating with Inmar and providing them with updates.

10:16 PM EST
Ed Folz (Software Developer) is pulled in to assist with the issue and to help determine why the Wave Management page is having issues.

10:28 PM EST
Ed helped determine what database query is being run by the Wave Management page and he saw evidence of slowness that Bastian needed a DBA to help address.

10:54 PM EST
Larry investigates the latest database issue

2:00 AM EST

Larry has made numerous changes to help fine-tune the database. However, there continue to issue processing the wave.

3:00 AM EST
Larry provided script updates so that Bastian could temporarily optimize the Wave Management page for the problem wave. This would allow them to perform every action need to close out the wave. After this was done and that wave was processed, Bastian would revert the script back to normal so that they could process everything else.

4:00 AM EST
Foster worked with the Inmar team and executed the scripts starting around 4 AM.

Root Cause:

The cause of this issue was a combination of a specific wave having a large number of orders and a database query being inefficient when running against that wave.

Resolution:

Larry provided script updates so that Bastian could temporarily optimize the Wave Management page for the problem wave. This would allow them to perform every action need to close out the wave. After this was done and that wave was processed, Bastian would revert the script back to normal so that they could process everything else.

Long term fix was the Database team has provided an update to one of our maintenance scripts to make it more efficient. This should prevent delete and archive from hanging up. Also, the Database team updated the view used by Wave Management so that it is optimized and runs faster.

Next Steps and Preventive Actions:

Continue monitoring database performance. 

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.