Knowledgebase Article Number
Published May 20th, 2020, at approximately 5:00 p.m. EST.
|At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on May 18th, 2020 described below.
On May 18th, 2020 a ticket was submitted to Bastian Solutions Support that touch screens on the Autostore at K&N were showing the message "Waiting on touch work". Ticket #143290 was generated. Shortly after the ticket was generated a call was received by Bastian Solutions Software Support. Brandon Frazier with K&N was calling in to address the ticket that was just generated. Below are the steps that were taken by Bastian Solutions while working on the problem.
May 18th, 2020
Sebastian Lynn, Support Specialist, with Bastian Solutions Software Support received the initial call from Brandon Frazier. Brandon stated that some ports were receiving a "waiting on touch work" message on some of the Autostore ports. Brandon also stated that this happened earlier in the day as well but did not contact support because the problem resolved itself in a few minutes. Sebastian stated he would get dialed into the system and start taking a look. He would reach back out when he had some more information.
Bastian Software Support Escalated to their Tier 2 resource to begin researching the issue.
Bastian Software Support reached back out to Brandon to discuss the situation. It was mentioned that 260 orders were in cubed status and pickers were picking but the number of cubed was not correlating to the number of tasks in Autostore. It was stated that this problem was intermittent and resolving itself at points. Pickers were still picking but some ports were receiving the message.
Bastian Software Support was able to pull numerous WCS communication errors from the application server. While troubleshooting they also pulled logs that showed many "bin errors" and errors assigning orders as seen below.
This leads the team to believe that they were dealing with communication issues with the Autstore itself. The team began looking into the database and applications that run the Autostore for problems.
Bastian Software Support reaches out to the Bastian Autostore support for confirmation on applications that run on the AS Controller. The team found 3 instances of the application AS Planner running on the AS controller. Only a single instance of this application should be running.
AutoStore Planner initiates AutoStore traffic based on commands and jobs (task queue, preparation queue,
Bastian Solutions requests to the K&N site that the Autostore needs to be stopped in order to fix this condition.
K&N Site stops the Autostore. Bastian Support closes all the AS Planner applications, ASDriver, and ASXhandler. Once these all close successfully they are all restarted in the proper sequence. The Autostore is then started by the K&N site team.
The rest of the week connected and monitored the area, without a single instance of this reoccurring. K&N Site confirmed all ports were operating and picking as normal.
Bastian Software Support received a new call from Brandon Frazier that all ports are now stating "waiting on touch work". Brandon stated that 156 orders were cubed in Exacta. This later turns out to be incorrect as it was orders in Exacta and not Autostore. Bastian Software Support, Foster Akado, was going to start looking into the situation again. All ports not picking or working.
Bastian Software Support started investigating certain orders for purposes of finding out why they were not delivering to ports.
Bastian Software Support escalated to Tier 3 and Management of the downtime situation happening at K&N
Bastian Solutions opens a bridge call for the support incident.
Bastian Escalates to involve Development resources. All Bastian Management has been contacted and all available resources are working the issue.
Bastian Solutions determines that there are 35 batches cubed that should be in Autostore but only 12 are showing in the Autostore Console. At this point, K&N is instructed to start picking on Port #1. Picking starts and runs as normal.
It is noted that only 3 task groups were available to be picked and work would run out soon. Bastian Software Solutions noticed that Work in cubed status needed to be resent to Autostore. Bastian Software Support resent the remaining work and it then created tasks in Autostore.
The team authorized picking to begin on Port #2. It was started and ran without a problem.
10:21 PM - 11:50 PM
The teams stayed on the bridge while picking continued to work correctly. At 11:50 it was noted that tasks in the Autostore were down to 12 and that is all they had to run for the night. Induction would run through 3rd shift and the morning shift on 5/19 would begin picking again. The bridge closed at 11:51 PM.
Next Steps and Preventive Actions:
Our commitment to you:
Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent re-occurrence.