Knowledgebase Article Number

360038068333

Description

Published October 29, 2019, at approximately 2 p.m. EST. 
Last updated on October 29, 2019.

Resolution 

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on August 22nd – August 23rd, 2019 described below.
 

Summary:

6:59 AM EST
The client notified Bastian that the Exacta applications were not accessible. 

7:50 AM EST
Bastian restarted All services and the entire App server. Applications were still inaccessible. 

8:00 AM EST
Escalation resources engaged on Bastian side and Management (Chase) notified as well. Escalated resource engaged Bastian DBA team. 

8:30 AM EST - 8:50 AM EST
Bastian resources identified database job and other long running database queries stuck and not progressing. Bastian turned off all exacta services including Portal,Mobile,App Services and Perfect Pick stations.

9:20 AM EST
Bastian resources stopped the stuck archiving process and re-enabled all Exacta Applications and Services. Site was able to begin picking. 

Root Cause:

Overnight SQL jobs were modified to include missed tables that had not been archived. Caused Archiving job to run over alloted time and impact database performance and applications.

Resolution:

Stopped archiving job and let the last step roll back.

Next Steps and Preventive Actions:

Bastian will create TFS Work Item and assign DBA team to create a promote for future archive changes to prevent this in the future.

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.