Knowledgebase Article Number

Description

February 14th, 2022, at approximately 12:10 p.m. EST. 

Resolution 

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on February 14th, 2022 described below.

Summary:

Monday, February 14th

12:10 PM EST

Kenny William called John Martin to get Software support engaged, and Eric reached out to Controls support to get both sides engaged. The issue that was reported was that nothing was diverting. Bastian Software Support coordinated with Controls Support to troubleshoot this issue. During the initial troubleshooting, Controls determined that they were not getting destinations from Software, but we could see destinations being sent to controls.

12:30 PM EST

We were able to determine that the issue causing the missed diverts was that the data was being sent late, which caused the diverting issues.

12:40 PM EST
The AOR software was restarted in an attempt to resolve the lag in data that was being sent to Controls. This did not resolve the issue, but at this time it was reported that some stuff was diverting correctly intermittently. 

12:45 PM EST
With the additional information that the containers were diverting intermittently, we began looking at the Database and were able to confirm that the issue was caused by database performance. Together with Kenny's assistance, we attempted to rebuild the indexes for the tables that were being used the most in the queries we identified.

1:05 PM EST
Software Support escalated the Database performance issue to the Bastian DBA team.

1:30 PM EST
Kenny identifies an Index that needs to be added to improve performance. The DBA identifies Database statistics that need to be created for the Document Line and Document Header tables to pair with the index that was created.

1:45 PM EST
Both changes are implemented and the issue from the software side has been resolved. The site notes that scan point 4 is still experiencing issues. Per Kenny, the issue was likely on the controls side. Troy reaches back out to controls and works with Brendan to resolve the issue with scan point 4.

2:20 PM EST
Controls resolve the issue with scan point 4 by removing a stuck tag in the PLC.

Root Cause:

The new query in the AOR logic was querying a column in the document line table that did not have an index or database statistics turned on.

Resolution:

The resolution was to add the index for Document Line and to create Statistics for the Document Line/Document Header tables. 

Next Steps and Preventive Actions:

Increase the scope of testing when it comes to implementing changes to test every aspect that we can.

Auto-create statistics to be turned on so they are not missing when a query requires them in the future.

Increase frequency of statistics rebuilds from weekly to NIGHTLY on job VDBA_CustomRebuildIndexes

Kept Exactadb Auto Update Statistics = FALSE

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service, and we assure you we're taking the required preventative measures to prevent reoccurrence.