Knowledgebase Article Number

 360048945273

Description

Published May 20th, 2020, at approximately 5:00 p.m. EST. 

Details

Summary: 

On May 18th, 2020 a ticket was submitted to Bastian Solutions Support that touch screens on the Autostore at K&N were showing the message "Waiting on touch work". Ticket #143290 was generated. Shortly after the ticket was generated a call was received by Bastian Solutions Software Support. Brandon Frazier with K&N was calling in to address the ticket that was just generated. Below are the steps that were taken by Bastian Solutions while working on the problem.

May 18th, 2020

2:11 PM

Sebastian Lynn, Support Specialist, with Bastian Solutions Software Support received the initial call from Brandon Frazier. Brandon stated that some ports were receiving a "waiting on touch work" message on some of the Autostore ports. Brandon also stated that this happened earlier in the day as well but did not contact support because the problem resolved itself in a few minutes. Sebastian stated he would get dialed into the system and start taking a look. He would reach back out when he had some more information.

3:08 PM

Bastian Software Support Escalated to their Tier 2 resource to begin researching the issue.

3:30 PM

Bastian Software Support reached back out to Brandon to discuss the situation. It was mentioned that 260 orders were in cubed status and pickers were picking but the number of cubed was not correlating to the number of tasks in Autostore. It was stated that this problem was intermittent and resolving itself at points. Pickers were still picking but some ports were receiving the message.

4:10 PM

Bastian Software Support was able to pull numerous WCS communication errors from the application server. While troubleshooting they also pulled logs that showed many "bin errors" and errors assigning orders as seen below. 

This leads the team to believe that they were dealing with communication issues with the Autstore itself. The team began looking into the database and applications that run the Autostore for problems.

4:40 PM

Bastian Software Support reaches out to the Bastian Autostore support for confirmation on applications that run on the AS Controller. The team found 3 instances of the application AS Planner running on the AS controller. Only a single instance of this application should be running.

AutoStore Planner initiates AutoStore traffic based on commands and jobs (task queue, preparation queue,
Port queues) from the Warehouse Management System (WMS).
In order to execute the planned activity, AutoStore Planner sends commands to AutoStore Driver.

5:05 PM

Bastian Solutions requests to the K&N site that the Autostore needs to be stopped in order to fix this condition.

5:15 PM

K&N Site stops the Autostore. Bastian Support closes all the AS Planner applications, ASDriver, and ASXhandler. Once these all close successfully they are all restarted in the proper sequence. The Autostore is then started by the K&N site team.

The rest of the week connected and monitored the area, without a single instance of this reoccurring. K&N Site confirmed all ports were operating and picking as normal.

6:16 PM

Bastian Software Support received a new call from Brandon Frazier that all ports are now stating "waiting on touch work". Brandon stated that 156 orders were cubed in Exacta. This later turns out to be incorrect as it was orders in Exacta and not Autostore. Bastian Software Support, Foster Akado, was going to start looking into the situation again. All ports not picking or working.

6:50 PM

Bastian Software Support started investigating certain orders for purposes of finding out why they were not delivering to ports.

8:00 PM

Bastian Software Support escalated to Tier 3 and Management of the downtime situation happening at K&N

8:08 PM 

Bastian Solutions opens a bridge call for the support incident.

8:45 PM

Bastian Escalates to involve Development resources. All Bastian Management has been contacted and all available resources are working the issue.

9:50 PM

Bastian Solutions determines that there are 35 batches cubed that should be in Autostore but only 12 are showing in the Autostore Console. At this point, K&N is instructed to start picking on Port #1. Picking starts and runs as normal.

10:13 PM

It is noted that only 3 task groups were available to be picked and work would run out soon. Bastian Software Solutions noticed that Work in cubed status needed to be resent to Autostore. Bastian Software Support resent the remaining work and it then created tasks in Autostore.

10:21 PM

The team authorized picking to begin on Port #2. It was started and ran without a problem.

10:21 PM - 11:50 PM

The teams stayed on the bridge while picking continued to work correctly. At 11:50 it was noted that tasks in the Autostore were down to 12 and that is all they had to run for the night. Induction would run through 3rd shift and the morning shift on 5/19 would begin picking again. The bridge closed at 11:51 PM.

Root Cause:

• In our investigation post the downtime we were able to determine from the AS controller logs that Windows updates were failing and caused the AS Planner Application to fail and restart 3X.

  • This was determined by contacting Autostore. They stated that this can cause multiple instances of AS Planner.

Resolution:

• Turn Off Windows updates per Autostore. Completed by Dave Strawser 5/19/2020 at 8:15 am
• Disable the Windows Update Service. Completed by Dave Strawser 5/20/2020 at 10:30 am.
• Update ASPlanner Application. Completed by Dave Strawser on 5/21/2020 at 3:10 pm.

Next Steps and Preventive Actions:

• Need to update ASDriver and ASXhandler ASAP. It can be done remotely.
• Need to update all remain AS Applications and Softwares per the K&N team approval.
• K&N requesting if it is possible for them to be alerted if the difference in exacta orders vs. tasks in Autostore is too high can they get an alert?
• K&N requesting if they can resend orders somehow if they see the out of sync condition.
• K&N is wanting to know if they can have readable access to our DB in order to pull there own data for dashboards?
• K&N would also like to revisit Bastian providing premium support as an option for them.

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent re-occurrence.