At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support.  Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on August 19th, 2021 described below. 

Summary: 

August 19

2:00 - 2:20 PM EST 

Parts Town called in to report that no work was coming to Exacta Touch ports and a "Lost connection to driver" error. While T1 Sarah Creasman resource was getting connected the issue seemed to have resolved itself.  

3:00 PM - 4:00 PM EST 

The site calls back to report that this is an ongoing issue. T1 resource Andrew Lynch confirms "Undefined and Connection errors as well as Exacta Touch Ports displaying a "DriverTimeout" message from AutoStore. Andrew runs the AS Connectivity Checker and confirms communication issues, this time with AutoStore Planner Check Log Publisher"  Andrew worked with onsite resources on the following steps. 

stopped Autostore via the ASconsole on the App server
stopped the Autostore processes on the AS controller PC: 10.10.161.21
In the following order: 
Stopped Xhandler
Stopped Xplanner
Stopped ASdriver

Checked task manager all have stopped

Started Xhandler
Started Xplanner
Started ASdriver

Andrew Lynch confirmed that the AS connectivity Checker no longer reported any errors. 

4:00 PM - 5:30 PM EST 

Some ports (13,14,16,18,19,20,21) begin receiving work. Other ports Still not receiving work, and AS bins seem to arrive at the port and immediately return to the AS grid. AS logging shows Exacta sending a Delete request for nonexistent work in Exacta with the following message

"Error while processing segment: Nonexisting Task 9540154: Task Delete" 

Bastian and Parts Town onsite resources coordinated a Full system restart for both Autostore and the Exacta Application Server (Where the AS-Backup, AS-Interface, and AS Log Publisher reside) that seemed to resolve all communication issues. Confirmed that Batches were now being sent to AS. 

August 30

Software Application Support Manager Travis Coleman engages T2 resource Andrew Downey to assist in determining a root cause for ticket 171428. 

Root Cause: 

If communication issues were present during the time Exacta attempts to send Batches of work to Autostore, AS may respond with "9999: DriveTimeout" message. Exacta would utilize the following logic

1. Call GetTaskInformation on the task group.
a. AutoStore returns the task group successfully -> we're good to go, treat the create task group as a success.
b. AutoStore returns an error (including the 9999 DriverTimeout) -> task group doesn't exist, treat the create task group as a failure (remove the task group from our containers so it will re-batch).

During the occurrence on Aug 19th,  We found that Autostore reported the DriverTimeout error back to Exacta as seen in Touch, and confirmed by resources on site. Although AutoStore responded with an error, the Batch did make it to the AS console. Exacta assuming the export failed, cleared the batch and queued the order to be re-batched. When AS brought the original Batch work to a port Exacta sends it away with a delete request because the task is not recognized. Once all "Ghost Batches" have been cleared during the continuous pick option in Exacta Touch, more work was able to flow from Exacta to Autostore. 

The initial reboot of the AutoStore applications on the console needed to include a restart of the AutoStore services that reside on the Exacta Application server. Once the Restart of both the AS applications and the Exacta Application Server (including the AS services installed there) was compelte, the communication issues were resolved. 

We have pulled the encypted AustoStore Logs and sent them to the AS Support team for further review. 

Our commitment to you: 

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service, and we assure you we're taking the required preventative measures to prevent reoccurrence.