Knowledgebase Article Number

360057105114

Description

Published Nov 13th, 2020, at approximately 2:15 p.m. EST. 

Details

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on between October 21st and November 8th, 2020 described below.

Summary: 

On Oct 21st, 2020 a support call was received by Support Specialist I Brent Smith, stating that bins were circulating but no work was being presented in Exacta Touch. While working with tier 2 support resources, Brent determined that there were numerous tasks in Autostore with no corresponding work in Exacta, and that the process of loading and then discarding these tasks was blocking valid work from making it to the ports.

Support removed these 'ghost' tasks as quickly as they could be identified.

We also determined at this time that several orders with the 'CART-' name format were having trouble in preprocessor. These orders were placed on hold to allow other work to enter the system.

On the morning of Oct 22nd Bastian Solutions development team provided a hotfix which causes PreProcessor to confirm a task group exists in Autostore before re-attempting the batch process, in order to prevent the creation of 'ghost' tasks. 

On October 23rd the issue of 'ghost' tasks was reported again. The Bastian Solutions development team confirmed the promote was working correctly and began reaching out to the Autostore team for additional assistance. 

It was determined that Exacta was now failing to submit new work similar to the issue with the CART orders from Oct 21. Bastian installed another promote to correct an issue with assignment logic that caused this late on the 23rd.

No major issues were reported again until Oct 29th. The 'ghost' tasks problem reappeared and was attributed to communication issues with Autostore. Bastian Support cleared the ghost tasks and reached out to Autostore Support again for help.

On Oct 30th Bastian received Autostore updates from Autostore support and installed these.

No major issues were reported again until the evening of Nov 9th, when preprocessor appears to have failed to send tasks to Autostore again. This was resolved by restarting the PreProcessor service, but we will need to capture and investigate another example more thoroughly in order to see if this was related to the assignment problem.

Root Cause:

• There are two issues in this ticket, one is the issue of 'ghost' tasks being created when timeouts occur in communication between Exacta and Autostore, and the other is PreProcessor being unable to advance to due inventory assignment issues. The first issue appears to be resolved. In the most recent reported trouble on Nov 9th, no 'ghost' tasks were created. However, this may have been a case of the same assignment issue we saw before, and we will need to monitor to ensure this does not occur again.

Resolution:

• Promote installed to make sure Exacta verifies batches were received by Autostore before creating new batches
• Promote installed to correct assignment issue 
• Autostore updates installed

Next Steps and Preventive Actions

• AdoreMe - continue to monitor and contact Bastian immediately should another instance of either ghost tasks or demand orders failing to assign be encountered
• Bastian - follow steps noted in ticket to investigate should these reoccur. 

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent re-occurrence.