Knowledgebase Article Number

4413676337047

Description

Published November 18th, 2021, at approximately 4:00 p.m. EST. 

Resolution 

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on November 18th, 2021 described below.

Summary:

Thursday, November 18th

3:52 PM EST
Jeremy Poe called into support to state he was having trouble started the AutoStore at the Parts Town facility.

3:54 PM EST

Call dropped

3:55 PM EST

Joe Doyle with Bastian Support called Jeremy Poe back. Was informed that the estop chain of the estop had tripped, and to resolve they had to press the SF reset button on the ASIO. They had not had to recover from this process in a long time and had forgotten about the SF reset button. Jeremy stated that no estop button had been pressed, so we speculated at that time that there could be a loose wire.

4:56 PM EST

Erick Cabral called into support to notify support that they were still having issues. Andres Garza answered the phone and attempted to pass the call to Joe Doyle, but the phone system failed the transfer. Joe Doyle connected to the AutoStore Service PC, and watched site reconnect to the Controller PC (after what appeared to be a Controller PC reboot) and restart AutoStore programs/services.

5:07 PM EST

Erick Cabral called back in to support and spoke with Mohammad Alwishah. Erick informed us that the issue they had was ASXHandler related. An error had occurred on Event Viewer of Console that indicated an issue with ASXhandler. When site investigated, they found ASXHandler service was locked up and would not stop/start/reset. This is what triggered the decision to restart the Controller PC.

7:46 PM EST

Alex Suarez downloaded a previous version of ASXHandler on the Controller PC. Alex advised Jeremy Poe of a known issue (AutoStore article regarding ASXHandler v1.2.5-4, dated 11/16/21) with Router and the ASXHandler service.  In talks with Parts Town site personnel is was determined that ASXhandler needed to be “rolled back” to a previous version. A window of time was set to proceed with the roll back.

10:55 PM EST

Felipe Terra called in to Jeremy with Parts Town to notify site that he was prepared to complete the rollback of ASXhandler. The new version was uninstalled, and the previous version installed.

11:02 PM EST

Felipe called Jeremy to inform him of the proper startup/shutdown procedure of the AutoStore programs/services.

Root Cause:

Article published by AutoStore regarding ASXHandler v1.2.5-4

Xhandler v1.2.5-4 issues

Nov 16, 2021•Knowledge

Xhandler v1.2.5-4 issues

Xhandler v1.2.5-4 has been removed from our Download Center due to issues that have been reported by our partners.
Xhandler v.1.2.2-4 is the version that is now available to download. 

The issues that have been discovered are the following:

• application crash when XHandler is started if receiving BIN_UPDATE before backup is downloaded
• issue where XHandler stops system after 12 minutes when waiting for superuser action in port
• application crash related to MOTOR_OVERLOAD_LIFT error
• application crash when receiving more than 10K robot notifies for a robot
• logic related to tracksensor validation on move errors needs to be improved

We are currently working on fixing the above issues. Once this is completed, we will run tests on our validations sites. We are hoping to have a new version ready next week. We will update this article then and inform you on the status.

Resolution:

 Rollback of ASXhandler, fixed the issue.

Next Steps and Preventive Actions:

-For systems that are currently running Xhandler v.1.2.5-4 - please downgrade to v.1.2.2-4
*to downgrade, please uninstall AutoStore Xhandler service using unins000 application located in AutoStore Xhandler folder. Then download v1.2.2-4 from Download Center and run the .exe file to start Xhandler (as version 1.2.2-4 does not run as service)

- For systems that are currently running v1.2.2-4 or earlier - please continue with the version you have installed today and upgrade when new Xhandler with a fix is released

We are sorry for the inconvenience this may have caused.

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service, and we assure you we're taking the required preventative measures to prevent reoccurrence.